diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..6d77ce7 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,45 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +ottyl turns a piped command's stdout (or a tailed file) into a local observability dashboard: searchable log timeline, ad-hoc metrics extracted from log lines via regex, and Grafana-style dashboards — all served from a single self-contained binary for the lifetime of that process. It can also poll a Prometheus/OpenMetrics endpoint alongside log ingestion. See `README.md` for the user-facing feature list and CLI usage. + +## Commands + +```bash +make build # builds webapp (npm install + vite build) then the release Rust binary; webapp/dist is embedded into the binary +make dev # frontend dev server with hot reload, proxying API calls to a separately running `cargo run` backend (see webapp/vite.config.ts) +make test # cargo test + frontend build (frontend has no test suite, just `tsc -b && vite build`) +make lint # cargo clippy --all-targets + oxlint (webapp) +make run # build then run ./target/release/ottyl +``` + +- Single backend test: `cargo test ` +- Backend tests live inline in each `src/*.rs` file (`#[cfg(test)] mod tests`), not in a separate `tests/` dir. +- The webapp has no unit tests — `npm run build` (tsc + vite) is the only frontend check besides lint. +- `webapp/dist` must exist before `cargo build` will produce a working binary (it's embedded via `fs_embed!`), so always build the frontend first — `make build` handles ordering; don't run `cargo build` alone after frontend changes without rebuilding `webapp/dist`. + +## Architecture + +Backend: Rust (axum + tokio), single binary, in `src/`. Frontend: React 19 + TypeScript + Tailwind + Recharts, in `webapp/`, built with Vite and embedded into the binary at build time (`src/api.rs`'s `WEBAPP_DIST` via `fs_embed!("webapp/dist")`). + +**Data flow:** `main.rs` wires everything together at startup — loads `Settings`, spawns the engine actor task, replays the active profile's param/metric declarations into it (before ingest starts, since there's no way to reprocess earlier lines later), then starts ingest (stdin/file tailing and/or Prometheus polling) and the axum HTTP server. + +- **`engine.rs`** (largest file, ~3400 lines) — the actor owning all live state: ingested log lines, declared params/metrics, computed metric buckets, alert evaluation. Runs as a single tokio task; all access is through `EngineHandle` sending `EngineMessage` over an mpsc channel and awaiting a response via oneshot — there is no shared mutable engine state accessed directly from request handlers. This is the file to read first to understand how declarations, filtering, and bucketing actually work. +- **`api.rs`** — axum routes and HTTP-facing `ApiState` (holds `EngineHandle` + `SettingsHandle`). Translates HTTP requests into `EngineMessage`s and DTOs back into JSON. Also serves the embedded webapp. +- **`settings.rs` / `profile.rs`** — persisted configuration. A **profile** is per-app: parsing rules (param/metric declarations) and dashboards, saved so they're there again next time that app runs through ottyl (see README's "Profiles"). Declarations are persisted here so the engine can replay them on restart; `api.rs` handlers that mutate engine state (e.g. `declare_metric`) call a corresponding persist method here so the two stay in sync — this dual-write is a recurring pattern (see `ApiState::persist_param_declaration` / `persist_metric_declaration` in `api.rs`), and adding a new declaration-mutating endpoint means updating both sides plus the load-time replay path. +- **`filter_parser.rs`** (grammar in `filter_parser.pest`, via `pest`) + **`filter_datetime.rs`** — parse the search/filter bar syntax (regex matches, metric comparisons like `latency_ms[value]<200`, relative/absolute datetimes). +- **`promql.rs`** — parses and evaluates the PromQL-style queries backing dashboard panels. +- **`logfmt.rs` / `ingest.rs`** — line ingestion: tailing stdin/a file, structured logfmt-style extraction, and polling a Prometheus/OpenMetrics `--metrics-url`. +- **`alerts.rs`** — alert rule evaluation and ntfy delivery. +- **`dto.rs`** — shared request/response types between `api.rs` and the frontend's `webapp/src/lib/api.ts`; **`data_value.rs`** / **`metric_primitive.rs`** — the value/type model for params and metrics; **`index_key.rs`** / **`rate.rs`** — bucket indexing and rate computation helpers. + +Frontend structure: `webapp/src/App.tsx` is the shell; `webapp/src/components/` has one component per feature area (`FilterBar`, `LogTimeline`, `Metrics`, `Params`, `Dashboards`, `Alerts`, `Docs`, `NavMenu`, ...) that map closely to the backend concepts above. `webapp/src/lib/api.ts` is the typed client for the Rust API; `webapp/src/lib/polling.ts` handles refresh polling; `webapp/src/lib/themes.ts` covers the style-preset/CSS system. + +## Conventions + +- **`tickets/`** holds scoped, written-up work items (problem statement, pointers to exact call sites with `file.rs:line` references, and an explicit scope section) — check here for planned/in-flight work before starting something that might overlap. Tickets are removed from this directory once implemented (see recent commit history — e.g. "Remove ticket 047 (implemented: ...)"). +- Declaration-related features (params, metrics) consistently need three coordinated changes: an `EngineMessage` variant + handler in `engine.rs`, a persistence method in `settings.rs`/`profile.rs`, and an `api.rs` route wiring them together (plus a webapp UI affordance). Grep for an existing similar declaration (e.g. how metric declaration is wired) as the template before adding a new one. +- Doc comments in this codebase tend to explain *why* a non-obvious choice was made (e.g. ordering constraints, a workaround, a past ticket's rationale) rather than restating what the code does — match that style rather than adding descriptive comments. diff --git a/WISHLIST.txt b/WISHLIST.txt new file mode 100644 index 0000000..e69de29 diff --git a/tickets/052-clear-params-and-reextract.md b/tickets/052-clear-params-and-reextract.md new file mode 100644 index 0000000..58fc6db --- /dev/null +++ b/tickets/052-clear-params-and-reextract.md @@ -0,0 +1,79 @@ +# 052 — Allow clearing all params, then re-extract them from already-ingested logs + +## Summary + +There is no bulk "clear params" action. The engine already supports +removing a single param declaration internally — +`EngineMessage::RemoveParamDeclaration { name, respond_to }` +(`src/engine.rs:85-92`), handled at `src/engine.rs:190-197` by removing +the `IndexKey::ParamDeclaration` entry — but it is only ever invoked from +`reload_engine_declarations` (`src/api.rs:664-682`), a profile-switch +cleanup path that filters out auto-derived params (comment at +`src/api.rs:665-668`) and removes the rest one at a time. There is no +route exposing this to the user, no persistence-side removal method +(`src/profile.rs` has `record_param_declaration` at `profile.rs:606` but +nothing to un-record it), and no webapp affordance +(`webapp/src/lib/api.ts:283-287` only exposes `getParams`/`declareParam`). + +The harder half of this request — "logs get re-iterated to extract +params from them" — is not just a rewording of `auto_declare_params`. +That function runs once, at ingest time, against each incoming line +(`src/engine.rs:154`, `auto_declare_params` defined at +`src/engine.rs:307-325`, backed by `extract_pairs` in `src/logfmt.rs`) and +is never replayed. Re-extraction after a clear means running that same +logic again, after the fact, against lines already sitting in the index. +This is structurally possible: raw ingested text is retained as +`IndexKey::IngestedLog { sequence, timestamp } -> DataValue::LogLiteral` +in the same `BTreeMap` (`src/engine.rs:148-152`), occupying the index +range between `IndexKey::Start` and `IndexKey::LogParamSplit` +(iterated at `src/engine.rs:1166`, `1181`) — but only for lines that +haven't already been evicted by `max_logs`/`evict_excess_logs` +(`src/engine.rs:151-156`). Lines evicted before a clear+re-extract cannot +regain their params; this bound should be surfaced to the user, not +silently ignored. + +Note `main.rs:75-96`'s `profile::load_into_engine` replay path is not +directly reusable here — it replays *declarations* into a fresh engine +before ingest starts, whereas this feature must re-run *extraction* +against logs the engine already holds. + +## Scope + +- Add a bulk "clear params" `EngineMessage` (or reuse + `RemoveParamDeclaration` in a loop, mirroring + `reload_engine_declarations`'s existing pattern at + `src/api.rs:664-682`) that removes all declared params — deciding + whether auto-derived and manually-declared params are both cleared or + only one category, and stating that choice in the PR (the existing + `reload_engine_declarations` precedent excludes auto-derived ones for a + reason worth re-reading before copying it here). +- Add a persisted-removal counterpart in `src/profile.rs`/`src/settings.rs` + so cleared params don't reappear from `settings.json` on next load. +- After clearing, re-run `auto_declare_params` against every retained + `IndexKey::IngestedLog` entry still in the index (oldest to newest, to + preserve declaration-order semantics), producing a fresh param set from + currently-available log text. +- Expose this as a single user-facing action (`DELETE /api/params` or + similar in `src/api.rs`, plus a button in whatever webapp component + hosts the params list) — not a per-param loop the user has to drive + from the UI. +- Surface to the user (via response body or UI copy) that re-extraction + only covers logs still retained in memory, not lines already evicted by + `max_logs`. + +## Acceptance criteria + +- Triggering "clear params" removes all param declarations from + `GET /api/params` and they do not reappear after a restart. +- Immediately after clearing, params re-appear derived from the log text + still held in the engine's index, without requiring new lines to be + ingested, verified with a seeded fixture where params are cleared and + re-extracted from a fixed set of already-ingested lines. +- If some lines have been evicted before the clear, the re-extracted + param set reflects only the retained subset, and this is not reported + as an error. + +## Depends on + +None (touches the same `RemoveParamDeclaration`/`reload_engine_declarations` +seam used by profile switching). diff --git a/tickets/053-move-sort-selector-to-timeline.md b/tickets/053-move-sort-selector-to-timeline.md new file mode 100644 index 0000000..19012a9 --- /dev/null +++ b/tickets/053-move-sort-selector-to-timeline.md @@ -0,0 +1,69 @@ +# 053 — Move the log sort selector from Log Preview to the Timeline + +## Summary + +The "Sort by" control lives in the wrong component. It's currently +rendered inside `LogPreview`'s own section header +(`webapp/src/components/LogPreview.tsx:327-336`) as a `SortPicker` +(`LogPreview.tsx:68-107`, a `