Update claude and tickets

tickets/052-clear-params-and-reextract.md

@@ -0,0 +1,79 @@
+# 052 — Allow clearing all params, then re-extract them from already-ingested logs
+
+## Summary
+
+There is no bulk "clear params" action. The engine already supports
+removing a single param declaration internally —
+`EngineMessage::RemoveParamDeclaration { name, respond_to }`
+(`src/engine.rs:85-92`), handled at `src/engine.rs:190-197` by removing
+the `IndexKey::ParamDeclaration` entry — but it is only ever invoked from
+`reload_engine_declarations` (`src/api.rs:664-682`), a profile-switch
+cleanup path that filters out auto-derived params (comment at
+`src/api.rs:665-668`) and removes the rest one at a time. There is no
+route exposing this to the user, no persistence-side removal method
+(`src/profile.rs` has `record_param_declaration` at `profile.rs:606` but
+nothing to un-record it), and no webapp affordance
+(`webapp/src/lib/api.ts:283-287` only exposes `getParams`/`declareParam`).
+
+The harder half of this request — "logs get re-iterated to extract
+params from them" — is not just a rewording of `auto_declare_params`.
+That function runs once, at ingest time, against each incoming line
+(`src/engine.rs:154`, `auto_declare_params` defined at
+`src/engine.rs:307-325`, backed by `extract_pairs` in `src/logfmt.rs`) and
+is never replayed. Re-extraction after a clear means running that same
+logic again, after the fact, against lines already sitting in the index.
+This is structurally possible: raw ingested text is retained as
+`IndexKey::IngestedLog { sequence, timestamp } -> DataValue::LogLiteral`
+in the same `BTreeMap` (`src/engine.rs:148-152`), occupying the index
+range between `IndexKey::Start` and `IndexKey::LogParamSplit`
+(iterated at `src/engine.rs:1166`, `1181`) — but only for lines that
+haven't already been evicted by `max_logs`/`evict_excess_logs`
+(`src/engine.rs:151-156`). Lines evicted before a clear+re-extract cannot
+regain their params; this bound should be surfaced to the user, not
+silently ignored.
+
+Note `main.rs:75-96`'s `profile::load_into_engine` replay path is not
+directly reusable here — it replays *declarations* into a fresh engine
+before ingest starts, whereas this feature must re-run *extraction*
+against logs the engine already holds.
+
+## Scope
+
+- Add a bulk "clear params" `EngineMessage` (or reuse
+  `RemoveParamDeclaration` in a loop, mirroring
+  `reload_engine_declarations`'s existing pattern at
+  `src/api.rs:664-682`) that removes all declared params — deciding
+  whether auto-derived and manually-declared params are both cleared or
+  only one category, and stating that choice in the PR (the existing
+  `reload_engine_declarations` precedent excludes auto-derived ones for a
+  reason worth re-reading before copying it here).
+- Add a persisted-removal counterpart in `src/profile.rs`/`src/settings.rs`
+  so cleared params don't reappear from `settings.json` on next load.
+- After clearing, re-run `auto_declare_params` against every retained
+  `IndexKey::IngestedLog` entry still in the index (oldest to newest, to
+  preserve declaration-order semantics), producing a fresh param set from
+  currently-available log text.
+- Expose this as a single user-facing action (`DELETE /api/params` or
+  similar in `src/api.rs`, plus a button in whatever webapp component
+  hosts the params list) — not a per-param loop the user has to drive
+  from the UI.
+- Surface to the user (via response body or UI copy) that re-extraction
+  only covers logs still retained in memory, not lines already evicted by
+  `max_logs`.
+
+## Acceptance criteria
+
+- Triggering "clear params" removes all param declarations from
+  `GET /api/params` and they do not reappear after a restart.
+- Immediately after clearing, params re-appear derived from the log text
+  still held in the engine's index, without requiring new lines to be
+  ingested, verified with a seeded fixture where params are cleared and
+  re-extracted from a fixed set of already-ingested lines.
+- If some lines have been evicted before the clear, the re-extracted
+  param set reflects only the retained subset, and this is not reported
+  as an error.
+
+## Depends on
+
+None (touches the same `RemoveParamDeclaration`/`reload_engine_declarations`
+seam used by profile switching).

tickets/053-move-sort-selector-to-timeline.md

@@ -0,0 +1,69 @@
+# 053 — Move the log sort selector from Log Preview to the Timeline
+
+## Summary
+
+The "Sort by" control lives in the wrong component. It's currently
+rendered inside `LogPreview`'s own section header
+(`webapp/src/components/LogPreview.tsx:327-336`) as a `SortPicker`
+(`LogPreview.tsx:68-107`, a `<select>` plus an "Fall back to ottyl_seq"
+checkbox), with its `LogSortSettings` type and
+`loadLogSortSettings`/`saveLogSortSettings` persistence helpers
+(`LogPreview.tsx:37-66`, localStorage key `ottyl.logPreview.sort`) also
+defined there. `LogTimeline.tsx` has no sort-related prop at all
+(`LogTimeline.tsx:44-56`) — its header only has time-range preset buttons
+and a "Clear selection" link (`LogTimeline.tsx:118-149`).
+
+The actual sort state is owned one level up, duplicated identically in
+`App.tsx`'s two top-level render modes: `FullApp`
+(`App.tsx:389-394`) and `StandaloneLog` (`App.tsx:263-268`), both via
+`useState<LogSortSettings>(loadLogSortSettings)` plus a
+`handleSortSettingsChange` callback. Both currently pass
+`sortSettings`/`onSortSettingsChange` only to `LogPreview`
+(`App.tsx:361`, `App.tsx:527`), never to `LogTimeline`
+(`App.tsx:237-243`, `354-360`/`520-526`). The setting already drives
+`fetchLogs` (`App.tsx:319-326`, `464-471`, dependency at `App.tsx:347`,
+`494`), which forwards `sortBy`/`sortMissingFallback` to
+`GET /api/logs` via `fetchLogs` in `webapp/src/lib/api.ts:242-262`. The
+backend already fully supports this — `src/dto.rs:86-90`'s
+`sort_by`/`sort_missing_fallback` fields, validated and applied in
+`src/engine.rs:1223-1281` (400 if `sort_by` isn't a declared param,
+`engine.rs:1227`; missing-value fallback behavior at `engine.rs:1260-1281`).
+The timeline query/DTO backing `fetchTimeline`
+(`webapp/src/lib/api.ts:264-274`) has no `sort_by` field and doesn't need
+one — this is a frontend-only relocation of where the control lives, not
+a backend change.
+
+## Scope
+
+- Move `SortPicker` (and ideally the `LogSortSettings` type plus
+  `loadLogSortSettings`/`saveLogSortSettings`) out of `LogPreview.tsx`
+  into `LogTimeline.tsx`, rendering it in the Timeline's header
+  (`LogTimeline.tsx:118-149`) alongside the existing range controls.
+- Add `sortSettings`/`onSortSettingsChange` props to `LogTimeline`.
+- Update both `App.tsx` render sites (`FullApp` around line 237,
+  `StandaloneLog` around line 354/520) to pass `sortSettings`/
+  `onSortSettingsChange` to `LogTimeline` instead of `LogPreview`.
+- `LogPreview` keeps rendering the `logs` prop as already sorted by the
+  parent's `fetchLogs` call — remove its own `SortPicker` usage and the
+  now-unused sort-control markup from its header, but keep consuming
+  `sortSettings` if it still needs read access to it (e.g. for a
+  "sorted by X" label), or drop that too if not needed.
+- No backend changes — `sort_by`/`sort_missing_fallback` on
+  `GET /api/logs` are unaffected.
+
+## Acceptance criteria
+
+- The "Sort by" selector and "fall back to ottyl_seq" checkbox appear in
+  the Timeline component's header, not in Log Preview's.
+- Changing sort order from the Timeline's control changes the order of
+  entries shown in Log Preview (same behavior as today, just relocated).
+- The chosen sort setting still persists across reloads (localStorage key
+  can be renamed if it moves file, but must not silently reset to
+  default on the first load after this change — migrate or fall back
+  gracefully).
+- `make build` (webapp `tsc -b && vite build`) passes with no leftover
+  unused imports/types in `LogPreview.tsx`.
+
+## Depends on
+
+None.

tickets/054-log-stream-format-detection.md

@@ -0,0 +1,63 @@
+# 054 — `log stream` output isn't recognized as a known format (folds into ticket 050)
+
+## Summary
+
+The report was "`log stream |& ottyl` should capture the format
+properly... tab separated format is being ignored, add it as a known
+type." Investigating `huig_dump.txt` (a real `log stream` capture already
+in the repo root) shows the premise needs correcting: the file is **not**
+tab-separated. `grep -P "\t"` finds a literal tab character on only 3 of
+697 lines, and none of those are the header or a representative data row
+— they're incidental (two `libinfo check path:` continuation lines, one
+WiFi stat line with an embedded tab mid-message). `od -c` on the header
+and data rows confirms the real structure is **space-padded fixed-width
+columns** (`Timestamp`, `Thread`, `Type`, `Activity`, `PID`, `TTL`,
+followed by a free-text `process: message` tail) — visually aligned
+columns that read as tab-separated but aren't.
+
+There is no TSV handling anywhere in the codebase today: `src/logfmt.rs`,
+`src/engine.rs`, and `src/ingest.rs` contain zero references to tabs or
+TSV. Format detection is entirely `detect_format` (`src/logfmt.rs:70-78`):
+try JSON, else fall back to `extract_pairs` (`logfmt.rs:45-59`) for
+logfmt, else `"none"` — no third branch exists, and `ottyl_fmt`
+(`src/engine.rs:1005`, filter compilation at `engine.rs:775`, `824`) only
+ever reports one of those three values.
+
+This is a duplicate of, not distinct from, ticket 050
+(`tickets/050-structured-header-extraction-and-bounded-logfmt-overfit.md`),
+which already identifies the exact same root cause — `log stream`
+output "is not logfmt at all — it's a fixed-column header... followed by
+a free-text `process: message` tail" — and already scopes a structured-
+header extractor plus the question of whether it becomes a new
+`ottyl_fmt` variant (e.g. `"log_stream"`). The reporter most likely
+visually misread the aligned whitespace columns as tab-separated.
+
+## Scope
+
+- Do not implement a separate "TSV `ottyl_fmt` type" — there is no
+  evidence of genuinely tab-delimited `log stream` output in this
+  codebase's fixtures, and building detection for a format that doesn't
+  occur would be speculative.
+- Close/merge this wishlist item into ticket 050's existing scope: its
+  structured-header extractor for `log stream`-shaped input already
+  covers "capturing the format properly" for this input shape.
+- If the user can supply a genuinely tab-delimited sample (e.g. from a
+  different `log stream --style` flag, or a shell pipeline that
+  reformats the output with real tabs), re-open this as a distinct
+  ticket scoped to that concrete input — file:line references above
+  should make it easy to confirm one way or the other before implementing
+  anything.
+
+## Acceptance criteria
+
+- No code changes result directly from this ticket; instead, ticket 050
+  is updated (or its implementation, when picked up, is confirmed) to
+  cover recognizing `log stream`'s fixed-column shape as a known format.
+- If a true TSV sample is later provided, a new ticket is filed against
+  it specifically, rather than reopening this one with an unconfirmed
+  premise.
+
+## Depends on
+
+050 (this ticket's concern is already inside 050's scope; treat as
+duplicate/fold-in rather than independent work).

tickets/055-ottyl-reserved-params-in-dashboards.md

@@ -0,0 +1,77 @@
+# 055 — Make `ottyl_*` reserved params usable in dashboard (PromQL) queries, not just filters
+
+## Summary
+
+Three reserved params exist, all hardcoded in the filter compiler:
+`ottyl_seq` (ingest sequence number), `ottyl_ts` (ingest timestamp,
+supports relative/absolute values), and `ottyl_fmt` (auto-detected
+`json`/`logfmt`/`none`) — defined at `src/engine.rs:773-775, 795, 806, 824`,
+grammar comments at `src/filter_parser.pest:7-9`, docs at
+`webapp/src/components/Docs.tsx:105-106, 375` ("reserved metric names
+present on every log line, so they're always filterable without
+declaring anything"). Alert conditions restrict to the same three plus
+`regex()`/`json[field]` (`src/alerts.rs:3, 199-225`).
+
+These already work end-to-end for the filter/query bar syntax used by
+`GET /api/logs`, `GET /api/timeline`, and alert conditions — that part of
+"accessible parameters for queries" is true today. What's missing is
+dashboards: PromQL-style panel queries (`src/promql.rs`,
+`evaluate_promql` in `src/engine.rs:1562-1601`) have no special case for
+`ottyl_*` anywhere. A PromQL query resolves `metric_name` via
+`bucket_values(index, &parsed.metric_name)`
+(`src/engine.rs`, called from `evaluate_promql`), which only returns data
+for names backed by a `DataValue::MetricDeclaration` and bucket entries
+created through `POST /api/metrics`. `ottyl_*` values are computed
+line-locally at filter-compile time and never written into declared-
+metric buckets, so `sum(ottyl_seq)` or similar in a dashboard panel would
+silently return an empty series rather than real data.
+
+Separately, `ottyl_*` never appear in `GET /api/params`
+(`src/api.rs:281-287` → `EngineQuery::ParamDeclarations` →
+`param_declarations_range`, `src/engine.rs:1340-1355`), since that only
+lists explicit `ParamDeclaration` index entries and `ottyl_*` never
+create one. `FilterBar.tsx` (`webapp/src/components/FilterBar.tsx:36-41`)
+has no autocomplete/suggestion list at all today (plain text input), so
+there's nothing currently excluding them from UI suggestions either —
+but if param suggestions are ever added, this gap should be kept in mind.
+
+## Scope
+
+- Add a special case in `evaluate_promql`/`src/promql.rs` (analogous to
+  the filter compiler's handling at `src/engine.rs:773-775`) that
+  synthesizes a virtual bucket series for `ottyl_seq`/`ottyl_ts`/
+  `ottyl_fmt` metric names from the ingested-log index, instead of
+  requiring a `MetricDeclaration`. Decide and document in the PR how
+  `ottyl_fmt`'s categorical string values map onto a numeric time
+  series (e.g. count-by-value, or reject it from PromQL numeric
+  aggregation with a clear error, since `ottyl_seq`/`ottyl_ts` are
+  naturally numeric but `ottyl_fmt` is not).
+- Decide whether `ottyl_*` should appear in `GET /api/params` (or a
+  parallel listing) so dashboard panel builders in the webapp can
+  discover them without prior knowledge of the reserved names; if yes,
+  extend `param_declarations_range` or add a small hardcoded merge at the
+  API layer rather than forcing them into the index as real
+  `ParamDeclaration` entries (they aren't user-declared and shouldn't be
+  deletable via ticket 051's/052's params-removal paths).
+- Update `README.md`/`Docs.tsx` to state whether `ottyl_*` is usable in
+  dashboard queries, once implemented (or explicitly document it as
+  filter-only, if the numeric/categorical mismatch above makes broad
+  PromQL support impractical for `ottyl_fmt`).
+
+## Acceptance criteria
+
+- A dashboard panel using a PromQL query over `ottyl_seq` or `ottyl_ts`
+  (e.g. `count(ottyl_seq)` or a rate over `ottyl_ts`) returns real data
+  from ingested logs, verified with a seeded fixture.
+- The behavior for `ottyl_fmt` in a PromQL query is either implemented
+  (with a stated value-to-series mapping) or explicitly rejected with a
+  clear, documented error — not a silent empty result.
+- `GET /api/params` behavior for `ottyl_*` is decided and documented;
+  if included, they're clearly marked as reserved/non-deletable rather
+  than indistinguishable from user-declared params.
+
+## Depends on
+
+None (independent of the params-removal tickets, but note the
+non-deletable requirement above if `ottyl_*` end up listed alongside
+regular params).

tickets/056-light-variants-for-remaining-colour-schemes.md

@@ -0,0 +1,77 @@
+# 056 — Add `Light *` variants for the remaining colour schemes
+
+## Summary
+
+Ticket 048 already added one light-surfaced variant, `light_green`
+(`webapp/src/lib/themes.ts:231-254`, commit `5371d3e`: "Adds a new
+'Light Green' scheme (light-surfaced, same green/magenta hue family as
+'Green') selectable alongside the existing built-ins"). The wishlist
+comment that prompted 048 — "we need a `Light Green` alongside `Green`"
+— is now clarified as applying to *every* colour scheme, not just green:
+the current built-in set is `dark`, `light`, `red`, `green`, `blue`,
+`yellow`, `pink`, `orange`, `light_green`
+(`webapp/src/lib/themes.ts:38-255`), so `red`, `blue`, `yellow`, `pink`,
+and `orange` still have no light-surfaced counterpart.
+
+Adding a scheme is not fully data-driven from `themes.ts` alone — six
+files must be kept in sync per scheme, per the no-build-step-links-them
+comment at `webapp/src/index.css:1-7`:
+
+1. `webapp/src/index.css:264-300` — the `html.light_green` CSS block is
+   the reference recipe: same hue + same accent hues as the dark
+   counterpart, but with `light`'s inverted (high-lightness)
+   surface/border/text scale, and `light`'s hue-neutral danger colors
+   (comment at `index.css:264-268` documents this explicitly).
+2. `webapp/src/lib/themes.ts:231-254` — the matching `BUILTIN_THEMES`
+   entry (used for CSS export/download and validation).
+3. `webapp/src/lib/api.ts:83-93` — the `ColorScheme` TS union type.
+4. `src/settings.rs:30-42` — the matching Rust `ColorScheme` enum variant,
+   with `#[serde(rename = "light_<name>")]`.
+5. `webapp/src/lib/useSettingsPanel.ts:24-34` — `THEME_CLASSES`, used to
+   apply/strip the `html.*` class.
+6. `webapp/src/components/NavMenu.tsx:20-31` — `COLOR_SCHEMES`, which
+   drives the picker UI; `schemeLabel()` (`NavMenu.tsx:34-35`) already
+   auto-titles `light_red` → "Light Red" by replacing underscores, so no
+   extra label wiring is needed once a name is added to this array.
+
+`PanelChart.tsx`'s multi-accent `color-mix()` series-color logic (touched
+by ticket 048) is generic per-theme rather than per-scheme-name, so it
+likely needs no changes — worth a quick check while implementing, not
+expected to require edits.
+
+Tickets 047/048's original files were deleted on implementation (git
+history only shows their removal commits, `5cb2dfd`/`7c331e7`; bodies
+aren't recoverable), but commit `5371d3e`'s message and the resulting
+`light_green` block are sufficient as the template.
+
+## Scope
+
+- Add `light_red`, `light_blue`, `light_yellow`, `light_pink`, and
+  `light_orange` following the exact `light_green` recipe: same base
+  hue and accent-hue choices as the corresponding dark-surfaced scheme,
+  `light`'s inverted surface/border/text lightness scale, `light`'s
+  neutral danger colors.
+- Update all six files listed above for each new scheme:
+  `webapp/src/index.css`, `webapp/src/lib/themes.ts`,
+  `webapp/src/lib/api.ts`, `src/settings.rs`,
+  `webapp/src/lib/useSettingsPanel.ts`, `webapp/src/components/NavMenu.tsx`.
+- Verify `schemeLabel()` produces correct titles ("Light Red", "Light
+  Blue", etc.) without additional label-mapping code.
+- Spot-check `PanelChart.tsx`'s series-color logic against one new light
+  scheme to confirm it needs no changes (or fix it if it does).
+
+## Acceptance criteria
+
+- All five new schemes are selectable in the settings picker
+  (`NavMenu.tsx`), persist correctly, and are downloadable as CSS via
+  the existing "download built-in theme" path.
+- Each new scheme's CSS block round-trips through `Settings`
+  serialization/deserialization (`src/settings.rs`) without errors,
+  verified with a test analogous to any existing `ColorScheme`
+  serialization test.
+- `make build` and `make lint` pass with the new schemes added.
+
+## Depends on
+
+048 (implemented; this ticket extends its established pattern to the
+remaining base hues).