Update claude and tickets

This commit is contained in:
Hugh Kaznowski
2026-09-06 09:30:38 +01:00
parent 2908cccd76
commit d17cfe9b83
7 changed files with 410 additions and 0 deletions
+45
View File
@@ -0,0 +1,45 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
ottyl turns a piped command's stdout (or a tailed file) into a local observability dashboard: searchable log timeline, ad-hoc metrics extracted from log lines via regex, and Grafana-style dashboards — all served from a single self-contained binary for the lifetime of that process. It can also poll a Prometheus/OpenMetrics endpoint alongside log ingestion. See `README.md` for the user-facing feature list and CLI usage.
## Commands
```bash
make build # builds webapp (npm install + vite build) then the release Rust binary; webapp/dist is embedded into the binary
make dev # frontend dev server with hot reload, proxying API calls to a separately running `cargo run` backend (see webapp/vite.config.ts)
make test # cargo test + frontend build (frontend has no test suite, just `tsc -b && vite build`)
make lint # cargo clippy --all-targets + oxlint (webapp)
make run # build then run ./target/release/ottyl
```
- Single backend test: `cargo test <test_name>`
- Backend tests live inline in each `src/*.rs` file (`#[cfg(test)] mod tests`), not in a separate `tests/` dir.
- The webapp has no unit tests — `npm run build` (tsc + vite) is the only frontend check besides lint.
- `webapp/dist` must exist before `cargo build` will produce a working binary (it's embedded via `fs_embed!`), so always build the frontend first — `make build` handles ordering; don't run `cargo build` alone after frontend changes without rebuilding `webapp/dist`.
## Architecture
Backend: Rust (axum + tokio), single binary, in `src/`. Frontend: React 19 + TypeScript + Tailwind + Recharts, in `webapp/`, built with Vite and embedded into the binary at build time (`src/api.rs`'s `WEBAPP_DIST` via `fs_embed!("webapp/dist")`).
**Data flow:** `main.rs` wires everything together at startup — loads `Settings`, spawns the engine actor task, replays the active profile's param/metric declarations into it (before ingest starts, since there's no way to reprocess earlier lines later), then starts ingest (stdin/file tailing and/or Prometheus polling) and the axum HTTP server.
- **`engine.rs`** (largest file, ~3400 lines) — the actor owning all live state: ingested log lines, declared params/metrics, computed metric buckets, alert evaluation. Runs as a single tokio task; all access is through `EngineHandle` sending `EngineMessage` over an mpsc channel and awaiting a response via oneshot — there is no shared mutable engine state accessed directly from request handlers. This is the file to read first to understand how declarations, filtering, and bucketing actually work.
- **`api.rs`** — axum routes and HTTP-facing `ApiState` (holds `EngineHandle` + `SettingsHandle`). Translates HTTP requests into `EngineMessage`s and DTOs back into JSON. Also serves the embedded webapp.
- **`settings.rs` / `profile.rs`** — persisted configuration. A **profile** is per-app: parsing rules (param/metric declarations) and dashboards, saved so they're there again next time that app runs through ottyl (see README's "Profiles"). Declarations are persisted here so the engine can replay them on restart; `api.rs` handlers that mutate engine state (e.g. `declare_metric`) call a corresponding persist method here so the two stay in sync — this dual-write is a recurring pattern (see `ApiState::persist_param_declaration` / `persist_metric_declaration` in `api.rs`), and adding a new declaration-mutating endpoint means updating both sides plus the load-time replay path.
- **`filter_parser.rs`** (grammar in `filter_parser.pest`, via `pest`) + **`filter_datetime.rs`** — parse the search/filter bar syntax (regex matches, metric comparisons like `latency_ms[value]<200`, relative/absolute datetimes).
- **`promql.rs`** — parses and evaluates the PromQL-style queries backing dashboard panels.
- **`logfmt.rs` / `ingest.rs`** — line ingestion: tailing stdin/a file, structured logfmt-style extraction, and polling a Prometheus/OpenMetrics `--metrics-url`.
- **`alerts.rs`** — alert rule evaluation and ntfy delivery.
- **`dto.rs`** — shared request/response types between `api.rs` and the frontend's `webapp/src/lib/api.ts`; **`data_value.rs`** / **`metric_primitive.rs`** — the value/type model for params and metrics; **`index_key.rs`** / **`rate.rs`** — bucket indexing and rate computation helpers.
Frontend structure: `webapp/src/App.tsx` is the shell; `webapp/src/components/` has one component per feature area (`FilterBar`, `LogTimeline`, `Metrics`, `Params`, `Dashboards`, `Alerts`, `Docs`, `NavMenu`, ...) that map closely to the backend concepts above. `webapp/src/lib/api.ts` is the typed client for the Rust API; `webapp/src/lib/polling.ts` handles refresh polling; `webapp/src/lib/themes.ts` covers the style-preset/CSS system.
## Conventions
- **`tickets/`** holds scoped, written-up work items (problem statement, pointers to exact call sites with `file.rs:line` references, and an explicit scope section) — check here for planned/in-flight work before starting something that might overlap. Tickets are removed from this directory once implemented (see recent commit history — e.g. "Remove ticket 047 (implemented: ...)").
- Declaration-related features (params, metrics) consistently need three coordinated changes: an `EngineMessage` variant + handler in `engine.rs`, a persistence method in `settings.rs`/`profile.rs`, and an `api.rs` route wiring them together (plus a webapp UI affordance). Grep for an existing similar declaration (e.g. how metric declaration is wired) as the template before adding a new one.
- Doc comments in this codebase tend to explain *why* a non-obvious choice was made (e.g. ordering constraints, a workaround, a past ticket's rationale) rather than restating what the code does — match that style rather than adding descriptive comments.
View File
+79
View File
@@ -0,0 +1,79 @@
# 052 — Allow clearing all params, then re-extract them from already-ingested logs
## Summary
There is no bulk "clear params" action. The engine already supports
removing a single param declaration internally —
`EngineMessage::RemoveParamDeclaration { name, respond_to }`
(`src/engine.rs:85-92`), handled at `src/engine.rs:190-197` by removing
the `IndexKey::ParamDeclaration` entry — but it is only ever invoked from
`reload_engine_declarations` (`src/api.rs:664-682`), a profile-switch
cleanup path that filters out auto-derived params (comment at
`src/api.rs:665-668`) and removes the rest one at a time. There is no
route exposing this to the user, no persistence-side removal method
(`src/profile.rs` has `record_param_declaration` at `profile.rs:606` but
nothing to un-record it), and no webapp affordance
(`webapp/src/lib/api.ts:283-287` only exposes `getParams`/`declareParam`).
The harder half of this request — "logs get re-iterated to extract
params from them" — is not just a rewording of `auto_declare_params`.
That function runs once, at ingest time, against each incoming line
(`src/engine.rs:154`, `auto_declare_params` defined at
`src/engine.rs:307-325`, backed by `extract_pairs` in `src/logfmt.rs`) and
is never replayed. Re-extraction after a clear means running that same
logic again, after the fact, against lines already sitting in the index.
This is structurally possible: raw ingested text is retained as
`IndexKey::IngestedLog { sequence, timestamp } -> DataValue::LogLiteral`
in the same `BTreeMap` (`src/engine.rs:148-152`), occupying the index
range between `IndexKey::Start` and `IndexKey::LogParamSplit`
(iterated at `src/engine.rs:1166`, `1181`) — but only for lines that
haven't already been evicted by `max_logs`/`evict_excess_logs`
(`src/engine.rs:151-156`). Lines evicted before a clear+re-extract cannot
regain their params; this bound should be surfaced to the user, not
silently ignored.
Note `main.rs:75-96`'s `profile::load_into_engine` replay path is not
directly reusable here — it replays *declarations* into a fresh engine
before ingest starts, whereas this feature must re-run *extraction*
against logs the engine already holds.
## Scope
- Add a bulk "clear params" `EngineMessage` (or reuse
`RemoveParamDeclaration` in a loop, mirroring
`reload_engine_declarations`'s existing pattern at
`src/api.rs:664-682`) that removes all declared params — deciding
whether auto-derived and manually-declared params are both cleared or
only one category, and stating that choice in the PR (the existing
`reload_engine_declarations` precedent excludes auto-derived ones for a
reason worth re-reading before copying it here).
- Add a persisted-removal counterpart in `src/profile.rs`/`src/settings.rs`
so cleared params don't reappear from `settings.json` on next load.
- After clearing, re-run `auto_declare_params` against every retained
`IndexKey::IngestedLog` entry still in the index (oldest to newest, to
preserve declaration-order semantics), producing a fresh param set from
currently-available log text.
- Expose this as a single user-facing action (`DELETE /api/params` or
similar in `src/api.rs`, plus a button in whatever webapp component
hosts the params list) — not a per-param loop the user has to drive
from the UI.
- Surface to the user (via response body or UI copy) that re-extraction
only covers logs still retained in memory, not lines already evicted by
`max_logs`.
## Acceptance criteria
- Triggering "clear params" removes all param declarations from
`GET /api/params` and they do not reappear after a restart.
- Immediately after clearing, params re-appear derived from the log text
still held in the engine's index, without requiring new lines to be
ingested, verified with a seeded fixture where params are cleared and
re-extracted from a fixed set of already-ingested lines.
- If some lines have been evicted before the clear, the re-extracted
param set reflects only the retained subset, and this is not reported
as an error.
## Depends on
None (touches the same `RemoveParamDeclaration`/`reload_engine_declarations`
seam used by profile switching).
@@ -0,0 +1,69 @@
# 053 — Move the log sort selector from Log Preview to the Timeline
## Summary
The "Sort by" control lives in the wrong component. It's currently
rendered inside `LogPreview`'s own section header
(`webapp/src/components/LogPreview.tsx:327-336`) as a `SortPicker`
(`LogPreview.tsx:68-107`, a `<select>` plus an "Fall back to ottyl_seq"
checkbox), with its `LogSortSettings` type and
`loadLogSortSettings`/`saveLogSortSettings` persistence helpers
(`LogPreview.tsx:37-66`, localStorage key `ottyl.logPreview.sort`) also
defined there. `LogTimeline.tsx` has no sort-related prop at all
(`LogTimeline.tsx:44-56`) — its header only has time-range preset buttons
and a "Clear selection" link (`LogTimeline.tsx:118-149`).
The actual sort state is owned one level up, duplicated identically in
`App.tsx`'s two top-level render modes: `FullApp`
(`App.tsx:389-394`) and `StandaloneLog` (`App.tsx:263-268`), both via
`useState<LogSortSettings>(loadLogSortSettings)` plus a
`handleSortSettingsChange` callback. Both currently pass
`sortSettings`/`onSortSettingsChange` only to `LogPreview`
(`App.tsx:361`, `App.tsx:527`), never to `LogTimeline`
(`App.tsx:237-243`, `354-360`/`520-526`). The setting already drives
`fetchLogs` (`App.tsx:319-326`, `464-471`, dependency at `App.tsx:347`,
`494`), which forwards `sortBy`/`sortMissingFallback` to
`GET /api/logs` via `fetchLogs` in `webapp/src/lib/api.ts:242-262`. The
backend already fully supports this — `src/dto.rs:86-90`'s
`sort_by`/`sort_missing_fallback` fields, validated and applied in
`src/engine.rs:1223-1281` (400 if `sort_by` isn't a declared param,
`engine.rs:1227`; missing-value fallback behavior at `engine.rs:1260-1281`).
The timeline query/DTO backing `fetchTimeline`
(`webapp/src/lib/api.ts:264-274`) has no `sort_by` field and doesn't need
one — this is a frontend-only relocation of where the control lives, not
a backend change.
## Scope
- Move `SortPicker` (and ideally the `LogSortSettings` type plus
`loadLogSortSettings`/`saveLogSortSettings`) out of `LogPreview.tsx`
into `LogTimeline.tsx`, rendering it in the Timeline's header
(`LogTimeline.tsx:118-149`) alongside the existing range controls.
- Add `sortSettings`/`onSortSettingsChange` props to `LogTimeline`.
- Update both `App.tsx` render sites (`FullApp` around line 237,
`StandaloneLog` around line 354/520) to pass `sortSettings`/
`onSortSettingsChange` to `LogTimeline` instead of `LogPreview`.
- `LogPreview` keeps rendering the `logs` prop as already sorted by the
parent's `fetchLogs` call — remove its own `SortPicker` usage and the
now-unused sort-control markup from its header, but keep consuming
`sortSettings` if it still needs read access to it (e.g. for a
"sorted by X" label), or drop that too if not needed.
- No backend changes — `sort_by`/`sort_missing_fallback` on
`GET /api/logs` are unaffected.
## Acceptance criteria
- The "Sort by" selector and "fall back to ottyl_seq" checkbox appear in
the Timeline component's header, not in Log Preview's.
- Changing sort order from the Timeline's control changes the order of
entries shown in Log Preview (same behavior as today, just relocated).
- The chosen sort setting still persists across reloads (localStorage key
can be renamed if it moves file, but must not silently reset to
default on the first load after this change — migrate or fall back
gracefully).
- `make build` (webapp `tsc -b && vite build`) passes with no leftover
unused imports/types in `LogPreview.tsx`.
## Depends on
None.
@@ -0,0 +1,63 @@
# 054 — `log stream` output isn't recognized as a known format (folds into ticket 050)
## Summary
The report was "`log stream |& ottyl` should capture the format
properly... tab separated format is being ignored, add it as a known
type." Investigating `huig_dump.txt` (a real `log stream` capture already
in the repo root) shows the premise needs correcting: the file is **not**
tab-separated. `grep -P "\t"` finds a literal tab character on only 3 of
697 lines, and none of those are the header or a representative data row
— they're incidental (two `libinfo check path:` continuation lines, one
WiFi stat line with an embedded tab mid-message). `od -c` on the header
and data rows confirms the real structure is **space-padded fixed-width
columns** (`Timestamp`, `Thread`, `Type`, `Activity`, `PID`, `TTL`,
followed by a free-text `process: message` tail) — visually aligned
columns that read as tab-separated but aren't.
There is no TSV handling anywhere in the codebase today: `src/logfmt.rs`,
`src/engine.rs`, and `src/ingest.rs` contain zero references to tabs or
TSV. Format detection is entirely `detect_format` (`src/logfmt.rs:70-78`):
try JSON, else fall back to `extract_pairs` (`logfmt.rs:45-59`) for
logfmt, else `"none"` — no third branch exists, and `ottyl_fmt`
(`src/engine.rs:1005`, filter compilation at `engine.rs:775`, `824`) only
ever reports one of those three values.
This is a duplicate of, not distinct from, ticket 050
(`tickets/050-structured-header-extraction-and-bounded-logfmt-overfit.md`),
which already identifies the exact same root cause — `log stream`
output "is not logfmt at all — it's a fixed-column header... followed by
a free-text `process: message` tail" — and already scopes a structured-
header extractor plus the question of whether it becomes a new
`ottyl_fmt` variant (e.g. `"log_stream"`). The reporter most likely
visually misread the aligned whitespace columns as tab-separated.
## Scope
- Do not implement a separate "TSV `ottyl_fmt` type" — there is no
evidence of genuinely tab-delimited `log stream` output in this
codebase's fixtures, and building detection for a format that doesn't
occur would be speculative.
- Close/merge this wishlist item into ticket 050's existing scope: its
structured-header extractor for `log stream`-shaped input already
covers "capturing the format properly" for this input shape.
- If the user can supply a genuinely tab-delimited sample (e.g. from a
different `log stream --style` flag, or a shell pipeline that
reformats the output with real tabs), re-open this as a distinct
ticket scoped to that concrete input — file:line references above
should make it easy to confirm one way or the other before implementing
anything.
## Acceptance criteria
- No code changes result directly from this ticket; instead, ticket 050
is updated (or its implementation, when picked up, is confirmed) to
cover recognizing `log stream`'s fixed-column shape as a known format.
- If a true TSV sample is later provided, a new ticket is filed against
it specifically, rather than reopening this one with an unconfirmed
premise.
## Depends on
050 (this ticket's concern is already inside 050's scope; treat as
duplicate/fold-in rather than independent work).
@@ -0,0 +1,77 @@
# 055 — Make `ottyl_*` reserved params usable in dashboard (PromQL) queries, not just filters
## Summary
Three reserved params exist, all hardcoded in the filter compiler:
`ottyl_seq` (ingest sequence number), `ottyl_ts` (ingest timestamp,
supports relative/absolute values), and `ottyl_fmt` (auto-detected
`json`/`logfmt`/`none`) — defined at `src/engine.rs:773-775, 795, 806, 824`,
grammar comments at `src/filter_parser.pest:7-9`, docs at
`webapp/src/components/Docs.tsx:105-106, 375` ("reserved metric names
present on every log line, so they're always filterable without
declaring anything"). Alert conditions restrict to the same three plus
`regex()`/`json[field]` (`src/alerts.rs:3, 199-225`).
These already work end-to-end for the filter/query bar syntax used by
`GET /api/logs`, `GET /api/timeline`, and alert conditions — that part of
"accessible parameters for queries" is true today. What's missing is
dashboards: PromQL-style panel queries (`src/promql.rs`,
`evaluate_promql` in `src/engine.rs:1562-1601`) have no special case for
`ottyl_*` anywhere. A PromQL query resolves `metric_name` via
`bucket_values(index, &parsed.metric_name)`
(`src/engine.rs`, called from `evaluate_promql`), which only returns data
for names backed by a `DataValue::MetricDeclaration` and bucket entries
created through `POST /api/metrics`. `ottyl_*` values are computed
line-locally at filter-compile time and never written into declared-
metric buckets, so `sum(ottyl_seq)` or similar in a dashboard panel would
silently return an empty series rather than real data.
Separately, `ottyl_*` never appear in `GET /api/params`
(`src/api.rs:281-287` → `EngineQuery::ParamDeclarations` →
`param_declarations_range`, `src/engine.rs:1340-1355`), since that only
lists explicit `ParamDeclaration` index entries and `ottyl_*` never
create one. `FilterBar.tsx` (`webapp/src/components/FilterBar.tsx:36-41`)
has no autocomplete/suggestion list at all today (plain text input), so
there's nothing currently excluding them from UI suggestions either —
but if param suggestions are ever added, this gap should be kept in mind.
## Scope
- Add a special case in `evaluate_promql`/`src/promql.rs` (analogous to
the filter compiler's handling at `src/engine.rs:773-775`) that
synthesizes a virtual bucket series for `ottyl_seq`/`ottyl_ts`/
`ottyl_fmt` metric names from the ingested-log index, instead of
requiring a `MetricDeclaration`. Decide and document in the PR how
`ottyl_fmt`'s categorical string values map onto a numeric time
series (e.g. count-by-value, or reject it from PromQL numeric
aggregation with a clear error, since `ottyl_seq`/`ottyl_ts` are
naturally numeric but `ottyl_fmt` is not).
- Decide whether `ottyl_*` should appear in `GET /api/params` (or a
parallel listing) so dashboard panel builders in the webapp can
discover them without prior knowledge of the reserved names; if yes,
extend `param_declarations_range` or add a small hardcoded merge at the
API layer rather than forcing them into the index as real
`ParamDeclaration` entries (they aren't user-declared and shouldn't be
deletable via ticket 051's/052's params-removal paths).
- Update `README.md`/`Docs.tsx` to state whether `ottyl_*` is usable in
dashboard queries, once implemented (or explicitly document it as
filter-only, if the numeric/categorical mismatch above makes broad
PromQL support impractical for `ottyl_fmt`).
## Acceptance criteria
- A dashboard panel using a PromQL query over `ottyl_seq` or `ottyl_ts`
(e.g. `count(ottyl_seq)` or a rate over `ottyl_ts`) returns real data
from ingested logs, verified with a seeded fixture.
- The behavior for `ottyl_fmt` in a PromQL query is either implemented
(with a stated value-to-series mapping) or explicitly rejected with a
clear, documented error — not a silent empty result.
- `GET /api/params` behavior for `ottyl_*` is decided and documented;
if included, they're clearly marked as reserved/non-deletable rather
than indistinguishable from user-declared params.
## Depends on
None (independent of the params-removal tickets, but note the
non-deletable requirement above if `ottyl_*` end up listed alongside
regular params).
@@ -0,0 +1,77 @@
# 056 — Add `Light *` variants for the remaining colour schemes
## Summary
Ticket 048 already added one light-surfaced variant, `light_green`
(`webapp/src/lib/themes.ts:231-254`, commit `5371d3e`: "Adds a new
'Light Green' scheme (light-surfaced, same green/magenta hue family as
'Green') selectable alongside the existing built-ins"). The wishlist
comment that prompted 048 — "we need a `Light Green` alongside `Green`"
— is now clarified as applying to *every* colour scheme, not just green:
the current built-in set is `dark`, `light`, `red`, `green`, `blue`,
`yellow`, `pink`, `orange`, `light_green`
(`webapp/src/lib/themes.ts:38-255`), so `red`, `blue`, `yellow`, `pink`,
and `orange` still have no light-surfaced counterpart.
Adding a scheme is not fully data-driven from `themes.ts` alone — six
files must be kept in sync per scheme, per the no-build-step-links-them
comment at `webapp/src/index.css:1-7`:
1. `webapp/src/index.css:264-300` — the `html.light_green` CSS block is
the reference recipe: same hue + same accent hues as the dark
counterpart, but with `light`'s inverted (high-lightness)
surface/border/text scale, and `light`'s hue-neutral danger colors
(comment at `index.css:264-268` documents this explicitly).
2. `webapp/src/lib/themes.ts:231-254` — the matching `BUILTIN_THEMES`
entry (used for CSS export/download and validation).
3. `webapp/src/lib/api.ts:83-93` — the `ColorScheme` TS union type.
4. `src/settings.rs:30-42` — the matching Rust `ColorScheme` enum variant,
with `#[serde(rename = "light_<name>")]`.
5. `webapp/src/lib/useSettingsPanel.ts:24-34` — `THEME_CLASSES`, used to
apply/strip the `html.*` class.
6. `webapp/src/components/NavMenu.tsx:20-31` — `COLOR_SCHEMES`, which
drives the picker UI; `schemeLabel()` (`NavMenu.tsx:34-35`) already
auto-titles `light_red` → "Light Red" by replacing underscores, so no
extra label wiring is needed once a name is added to this array.
`PanelChart.tsx`'s multi-accent `color-mix()` series-color logic (touched
by ticket 048) is generic per-theme rather than per-scheme-name, so it
likely needs no changes — worth a quick check while implementing, not
expected to require edits.
Tickets 047/048's original files were deleted on implementation (git
history only shows their removal commits, `5cb2dfd`/`7c331e7`; bodies
aren't recoverable), but commit `5371d3e`'s message and the resulting
`light_green` block are sufficient as the template.
## Scope
- Add `light_red`, `light_blue`, `light_yellow`, `light_pink`, and
`light_orange` following the exact `light_green` recipe: same base
hue and accent-hue choices as the corresponding dark-surfaced scheme,
`light`'s inverted surface/border/text lightness scale, `light`'s
neutral danger colors.
- Update all six files listed above for each new scheme:
`webapp/src/index.css`, `webapp/src/lib/themes.ts`,
`webapp/src/lib/api.ts`, `src/settings.rs`,
`webapp/src/lib/useSettingsPanel.ts`, `webapp/src/components/NavMenu.tsx`.
- Verify `schemeLabel()` produces correct titles ("Light Red", "Light
Blue", etc.) without additional label-mapping code.
- Spot-check `PanelChart.tsx`'s series-color logic against one new light
scheme to confirm it needs no changes (or fix it if it does).
## Acceptance criteria
- All five new schemes are selectable in the settings picker
(`NavMenu.tsx`), persist correctly, and are downloadable as CSS via
the existing "download built-in theme" path.
- Each new scheme's CSS block round-trips through `Settings`
serialization/deserialization (`src/settings.rs`) without errors,
verified with a test analogous to any existing `ColorScheme`
serialization test.
- `make build` and `make lint` pass with the new schemes added.
## Depends on
048 (implemented; this ticket extends its established pattern to the
remaining base hues).