- 076: gate auto-declared log-stream column/message params
(`ParamDeclaration::log_stream_gated`) behind `is_log_stream_line` at
extraction time, since their fixed-offset regex has no content anchor
and previously extracted garbage from continuation lines and the
header row itself (the regex crate doesn't support lookaround, so
this couldn't be baked into the regex text).
- 074: end-to-end regression test confirms sorting a real `log stream`
capture by a log-stream column (e.g. PID) now actually reorders
results, since 076's fix removes the garbage values that were
corrupting the sort helper index; also add a "Building sort
index…" indicator in the timeline UI for a slow first-time sort.
- 072: attach the reserved `ottyl_fmt` pseudo-param to each
`LogEntryDto`'s params so it can be picked as a Log Preview column
(previously listed as pickable but always blank).
- 073: track a rolling 5s high-watermark of ingestion-to-engine-processing
lag and surface it in the nav stats popover next to approx_bytes_total.
- 071: group the regex(...) help button back with the filter bar's other
controls, and restructure the Filtering docs section into scannable
per-feature subsections (also fixes stale log_type/process/pid mention).
Tickets 070 (CLI scoping) and 075 (API unresponsiveness investigation)
are left in place per their own scope — both are explicitly
conversation/investigation tasks, not implementation tickets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
evaluate_promql previously resolved every metric name through bucket_values,
which only returns data for a real MetricDeclaration — so a dashboard panel
querying ottyl_seq/ottyl_ts/ottyl_fmt (already supported in the filter bar
and alert conditions) silently returned an empty series. ottyl_seq/ottyl_ts
are now synthesized directly from the ingested-log index instead of requiring
a declaration; ottyl_fmt is categorical (json/logfmt/none) with no sensible
numeric reading, so it's rejected up front with a clear error pointing back
at the filter bar rather than returning nothing.
GET /api/params now also lists the three reserved names (marked `reserved:
true`, distinct from `auto_derived`) purely for discoverability — merged in
at the API layer only, not the engine's internal ParamDeclarations query, so
profile-switch removal and metric-param-promotion logic are unaffected.
declare_param also now rejects declaring a param under one of these names.
Docs.tsx and the Params panel are updated to reflect the new dashboard
support and the reserved/non-editable listing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
worktree-agent-a19b9ef959dcda6b2 (ticket 049) was branched from before "Auto
derived format" (1a481d3) landed on main, so its synchronous
persist_metric_declaration(_update) and ProfileStore::record_metric_declaration(_update)
didn't know about promoting auto-derived engine params into the persisted
profile. This merge reconciles the two:
- src/api.rs: kept the async persist_metric_declaration/_update methods and
promoted_param_records helper from "Auto derived format", but switched their
persistence from ProfileStore::load/record_metric_declaration to the new
Settings-based settings.record_metric_declaration(_update), matching ticket
049's single-settings.json model. Dropped the now-unused
active_profile_location helper in favor of locking self.settings directly.
- src/profile.rs: removed the standalone ProfileStore struct/impl entirely
(049 folds profiles into Settings), keeping load_legacy_profile_store for
migrating a pre-049 profiles.json. Extended record_metric_declaration and
record_metric_declaration_update (now methods on Settings) to keep taking
promoted_params: &[ParamDeclarationRecord], persisting promoted auto-derived
params alongside the metric via Settings::save. Moved the ticket 037/038
legacy-shape migration (migrate_legacy_metric_params /
migrate_duration_value_params) out of the removed ProfileStore::load into a
new pub(crate) migrate_legacy_profile_shapes(profiles), invoked from
Settings::load (via a new Settings::migrate_legacy_profile_shapes) so
profiles inlined in settings.json still get migrated forward on every load.
Rewrote the ProfileStore-based tests (legacy-shape migration tests, the
no-op-migration test, and the auto-derived-param promotion round-trip test)
to exercise Settings directly, including a write_legacy_settings_json helper
that replaces the old write_legacy_profiles_json.
- src/api.rs tests: rewrote declaring_metric_against_auto_derived_param_promotes_it_into_the_profile
to create/activate the profile via settings_handle.lock() and reload via
Settings::load instead of ProfileStore.
Also fixed one clippy::collapsible_if warning introduced by the new
Settings::migrate_legacy_profile_shapes.
cargo build, cargo test (222 passed, 0 failed, 0 ignored), and
cargo clippy --all-targets all pass; no new clippy warnings beyond
pre-existing ones in files untouched by this merge.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
/api/logs now accepts sort_by (an already-declared param name) and
sort_missing_fallback: lines carrying that param are ordered by its value,
while lines missing it are either appended in ottyl_seq order (fallback,
the default) or dropped from the response. Default ottyl_ts-tracking
order is unchanged when sort_by is omitted.
The log viewer (both the inline panel and its popout) exposes this as a
"Sort by" control next to the column picker, persisted to localStorage
under one shared key (ottyl.logPreview.sort) — the same global-viewer-
setting convention already used for visible param columns and the log
viewer's height, since there's no per-dashboard scope for this to divide
by.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Fold ProfileStore's separate profiles.json into Settings itself, so
settings.json is the single source of truth instead of two files kept
in sync by hand. Settings now carries `profiles: Vec<Profile>` and
`active_profile: Option<String>` directly; every former ProfileStore
mutation (create, delete, set_active, record_param_declaration,
record_metric_declaration, dashboard CRUD, ensure_default) is now a
method on Settings that persists via Settings::save.
A custom Deserialize (via SettingsOnDisk) keeps an old settings.json
(with the removed Profiles pointer struct) loading cleanly, and
Settings::load runs a one-time migration: if profiles is empty and a
legacy <settings_dir>/profiles/profiles.json exists, its contents are
merged in, settings.json is resaved, and the old file is renamed so it
isn't re-imported.
Updated all call sites in api.rs and main.rs to the single-path API,
and ported profile.rs's ProfileStore tests to exercise Settings
directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Previously a Value-role param with kind: Duration silently meant "elapsed
time since the previous same-labeled event" instead of parsed text — the
same MetricPrimitiveKind::Duration variant meant two unrelated things
depending on which list (label_params vs value_param) referenced it.
Add ParamDeclaration::elapsed_since_last, a third extraction source
alongside capture_group/json_field_path: when set, the param is computed
(via a new resolve_param_value, used uniformly for label_params and
value_param) rather than text-extracted, and declare_param validates it's
paired with kind: Duration. apply_metric_to_log no longer branches on
(role, kind) — it dispatches purely on what each resolved param declares.
Migrate existing profiles: migrate_duration_value_params converts any
Duration-kind param referenced as a metric's value_param into the new
extractor, and resolve_or_create_legacy_param (ticket 037's pre-existing
migration) now carries the same meaning forward for pre-037 inline
Value-role Duration matchers.
Params.tsx offers "Elapsed since previous event" as its own Extraction
option (hides the now-irrelevant message-match-regex/capture-group
fields and forces kind to duration); Metrics.tsx labels such params
accordingly wherever they're listed as label/value param choices.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Collapse the Params-panel ParamDeclaration and the metric-embedded
RegexToMetricPrimitiveMatcher into a single ParamDeclaration type
(name, regex/capture-group-or-json-field-path, kind). MetricDeclaration
now references params by name via label_params: Vec<String> and
value_param: Option<String> instead of embedding its own extractor per
param, so a param declared once in the Params panel can be reused as a
label or value by any number of metrics. Declaring a metric that
references an undeclared param name is now rejected at declare-time.
Backend:
- src/data_value.rs: unified ParamDeclaration struct; MetricDeclaration
data value now carries label_params/value_param instead of a Vec of
inline matchers.
- src/engine.rs: DeclareParamRequest is now an alias for
ParamDeclaration; DeclareMetricRequest carries label_params/
value_param; validate_metric_request checks referenced param names
against the declared-param store; apply_metric_to_log resolves
label_params/value_param against a per-ingest resolved-param map
instead of reading an inline matcher list.
- src/profile.rs: MetricDeclarationRecord carries label_params/
value_param plus a private, serialize-skipped legacy_params field
that only ever gets populated by deserializing a pre-037 profile
file's old "params" key. ProfileStore::load migrates any such legacy
metric into the unified shape (reusing an existing same-shaped bare
param by name, or creating one implicitly named
"<metric_name>.<param_name>"), then persists the migrated store so
the on-disk file only carries the legacy shape once. Migration is
idempotent (deterministic naming) and covered by dedicated tests.
- src/api.rs / src/dto.rs: MetricDeclarationDto and the declare/update
metric response bodies updated to the new shape.
Frontend:
- webapp/src/lib/api.ts: MetricDeclaration type now has label_params/
value_param; MetricParamMatcher/ParamRole types removed.
- webapp/src/components/Metrics.tsx: the declaration form now fetches
declared params and lets you check off label params / pick a value
param from that list, instead of re-entering a regex/capture-group/
kind per metric param.
Leaves ticket 038 (duration as a derived param kind) and 039 (param
query resolving declared params) untouched, as scoped.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Removes the "Use as theme" checkbox/is_theme distinction from
NavMenu.tsx's CSS import and link forms — every imported/linked
stylesheet is now always validated against the canonical --c-*
variable set and becomes a selectable theme, same as today's
is_theme:true path. Drops is_theme entirely from the backend
(CssEntry, ImportCssRequest, LinkCssRequest, CssEntryDto) and from
the frontend CssEntry type; an existing settings.json with a stored
is_theme:false entry still deserializes fine (the field is simply
unknown/ignored) and that entry becomes selectable like any other,
so no migration step is needed. Also drops the now-dead
always-on-overlay CSS injection path from useSettingsPanel.ts, and
changes the CSS filename placeholder from "name.css" to "enter name".
Implemented directly against main rather than merging the dispatched
worktree branch for this ticket, since that branch's base predated
ticket 012 (the custom-themes/imported-CSS consolidation) and its
NavMenu structure no longer matched current main.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
RegexToMetricPrimitiveMatcher and the standalone param declaration
(DeclareParamRequest/DataValue::ParamDeclaration) gain an optional
json_field_path, an alternative to capture_group that reads a
dot-separated field path (e.g. "foo.bar") out of lines that parse as
JSON objects. It's additive and serde-defaulted, so existing
regex-based params/metrics are unaffected. Params.tsx now offers a
"JSON field" extraction option when the message match regex looks
like a JSON object.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Tracks a running total of ingested log bytes (approximated as each
log's raw UTF-8 text length), incremented on ingest and decremented
in evict_excess_logs when a log is dropped. Exposed via StatsDto's
new approx_bytes_total field on GET /api/stats, and displayed in
NavMenu's stats panel formatted with toLocaleString() (comma
separators).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Alert rules no longer carry their own ntfy topic/server; Settings now
holds one process-wide ntfy sink (with its own "send test notification"),
and the Alerts panel is left to just define rule conditions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Pass over src/*.rs, webapp/src/**, and the in-app "?" docs prose:
condensed multi-paragraph doc comments to single lines that state the
non-obvious why, deleted comments that only restated adjacent code, and
tightened the Docs.tsx help-panel wording. No logic, structure, or code
formatting changed — verified via cargo test (143 passed) and
tsc -b && vite build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Themes download as a small `:root { --c-*: ...; }` CSS file (full
canonical variable set); uploading validates that set is complete and
registers the file as a CssEntry the theme picker can select. A new
`is_theme` flag on CssEntry distinguishes theme entries from arbitrary
imported/linked CSS so they don't leak into the page unconditionally
like the existing "Configured CSS" list does — a selected custom
theme's CSS is injected only while active, with built-in html.<scheme>
classes removed so its :root override isn't outranked in the cascade.
Linked (URL) custom themes reuse the existing CSS link mechanism and
inherit its non-deletable semantics.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
New Alerts panel: lists rules with condition summaries and an enabled
toggle, and a condition-type-specific editor (Immediate/Threshold/NewValue/
Ratio) with client-side validation before save, reusing FilterBar's query
input styling for query-based conditions. Adds a test-notification endpoint
(POST /api/settings/alerts/test) so an ntfy topic/server can be verified
without saving a rule first, plus a docs section and help icons matching
ticket 021's pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Adds AlertRule/AlertCondition/AlertSink models with CRUD persisted on
Settings (mirroring CssEntry's shape), per-line trigger evaluation hooked
into the engine's ingest path (Immediate/Threshold/NewValue/Ratio
conditions), and best-effort ntfy delivery spawned off the ingest hot path.
Design notes:
- Alert queries reuse FilterBar syntax but only support the
regex()/sequence/timestamp subset (no metric[param] comparisons), since
resolving those would require a full bucket rescan per alert per ingested
line rather than cost proportional to new logs.
- Ratio's zero-denominator case skips evaluation for that line rather than
treating the ratio as infinite.
- NewValue's seen-value set is in-memory only and does not survive a
restart, matching the engine's other non-persisted runtime state.
engine::run now takes a SettingsHandle (moved from api.rs to settings.rs so
engine.rs can depend on it without a circular import) to read the current
alert rules on each ingest.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
ensure_default now seeds a brand-new Default profile (not one that
already existed) with a "log_category" counter metric and an
"Overview" dashboard containing a line and a pie panel, both querying
that metric so each renders both series from a single PromQL query
like any user-created panel.
The counter classifies every line into exactly "category=error" or
"category=other" via two new fixed-label fields on metric
declarations: DeclareMetricRequest::matched_labels/unmatched_labels.
These apply literal label values instead of parsing captured text,
which is what keeps a case-insensitive match (e.g. "ERROR" vs "Error")
collapsed into one clean series instead of fragmenting by source
casing, and is also the only way to express the "line does NOT match"
side at all: plain regex alternation can't do it without lookaround,
which the `regex` crate doesn't support (a catch-all alternation
branch always wins at the search's leftmost position). Both fields
default to None and are backward compatible with existing profiles.
Seeded panels are ordinary editable/deletable dashboard state,
persisted through the normal profile save path; ensure_default never
re-seeds or duplicates them on a later call once the profile exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Settings.max_logs (default None = unlimited) is pushed to the engine
via a new EngineMessage::SetMaxLogs, enforced on every ingest and
whenever the setting changes at runtime (no restart required).
Eviction drops the oldest (lowest-sequence) logs; sequence numbering
isn't renumbered and buckets already derived from an evicted log are
left in place. Exposed via PATCH /api/settings (max_logs: number |
null) and a NavMenu numeric field (blank = unlimited).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Backend: replace Profile.dashboards' opaque JSON stub with typed
Dashboard/Panel structs, add ProfileStore CRUD (create/save/list/delete
dashboard) and matching /api/settings/profiles/:name/dashboards[/:id] routes,
with tests covering the HTTP round trip and persistence across a simulated
restart.
Frontend: add recharts (lightweight, React 19-compatible) and a Dashboards
component with a dashboard builder (create dashboards, add/remove/edit panels
with a PromQL query + chart type), live per-panel previews that respect the
active time range and show malformed-query errors inline instead of crashing,
and load/save wired to the new profile-backed persistence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Adds thin HTTP routes for profile CRUD and colour-scheme updates
(GET/PATCH /api/settings, GET/POST /api/settings/profiles,
DELETE /api/settings/profiles/:name, POST /api/settings/profiles/:name/activate)
following the existing CSS-route pattern in api.rs, with activate hot-swapping
the running engine's declared params/metrics so switching profiles needs no
restart. Builds out the NavMenu frontend with Settings (colour scheme picker,
CSS import/list/delete) and Presets (list/create/switch/delete profiles with
declaration counts) panels, and a useSettingsPanel hook that applies the
active colour scheme and imported CSS to the document live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Adds a filter evaluator in engine.rs that compiles a parsed FilterExpr
against the current index once per query (regex compiled once,
sequence/timestamp compared directly against IngestedLog fields,
other metric comparisons resolved to a HashSet<u64> of matching
sequences via a bucket scan) and evaluates it per log line with no
further index access. Wires this into EngineQuery::Logs/Timeline via
a new LogsParams::filter/TimelineParams::filter field, and adds
EngineResponse::Error so a malformed filter string surfaces as a
query-level error (mapped to 400 in api.rs).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Hand-rolls a minimal PromQL parser (src/promql.rs) for metric-name +
label-matcher selection, optional [range] selectors, and top-level
sum()/avg()/rate() wrapping, rather than pulling in a full PromQL parser
crate — the required subset is small and a hand-rolled parser stays fully
engine-independent (mirrors filter_parser.rs) with no unused-AST overhead.
Adds EngineQuery::Promql/EngineResponse::Promql (engine parses + evaluates
in place against the index, no copying) and GET /api/query, returning
Vec<PromqlSeriesDto> (labels + (timestamp_ms, f64) points) for charting,
with malformed queries surfaced as 400 + ErrorDto.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Extend GET /api/logs to derive each declared param's value per log line
(regex + capture group + kind, matching that line's text) and include it as
a params map on LogEntryDto, omitting entries that don't match rather than
sending nulls. Add GET /api/params (list_params, mirroring list_metrics) so
the frontend can discover currently-declared param names. In LogPreview, add
a column-picker popover backed by that list, with the selection persisted in
localStorage under `ottyl.logPreview.visibleParamColumns`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Exposes param/metric declarations and derived bucket data over HTTP:
POST /api/params and POST /api/metrics validate and declare against the
engine, persisting into the active profile (if any) so declarations survive
a restart; GET /api/metrics lists declarations; GET /api/metrics/:name
queries bucket values with label, time-range, sequence, and limit filters.
Adds a new EngineQuery::BucketValues variant that layers this filtering on
top of ticket 003's bucket_values helper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Wires up Settings.css_entries behind a small settings API: import raw CSS
content to a file stored alongside settings.json, link external CSS
paths/URLs by reference (never deletable/copied), list all entries, delete
imported entries, and serve back imported content. CSS storage logic lives
on Settings (import_css/link_css/remove_css/css_content) with api.rs
handlers staying thin, matching the existing engine-query pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Adds IndexKey::MetricDeclaration handling: DeclareMetric/RemoveMetricDeclaration
engine messages and a MetricDeclarations query, mirroring the pattern from ticket
001's param declarations. Params are derived from inline named capture groups on
the metric's own regex for v1 (referencing existing ParamDeclarations is left for
a future ticket). Validates the regex and every RegexToMetricPrimitiveMatcher's
capture-group reference at declaration time, reporting all bad matchers at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Implements the ParamDeclaration slice of the primary index: DataValue::ParamDeclaration
holds a regex + named capture group + MetricPrimitiveKind, stored under
IndexKey::ParamDeclaration{name} between LogParamSplit and ParamMetricSplit.
EngineMessage gains DeclareParam (validates the regex compiles and the capture
group exists before storing, returning a clear error otherwise) and
RemoveParamDeclaration; EngineQuery gains ParamDeclarations to list them back out
as ParamDeclarationDto. No API wiring yet (separate ticket).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
