The engine actor and API handlers previously called left_right's
WriteHandle/ReadHandleFactory methods (append/publish/enter/handle)
directly. IndexWriter now exposes mutate()/peek() and IndexReader
exposes handle() (returning an IndexReadHandle with its own enter()),
so how the single-writer/multi-reader index is accessed is named and
scoped to these two types instead of being raw left_right calls
scattered through engine.rs/api.rs/profile.rs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
Profile (legacy-shape migration, load_into_engine, load_legacy_store),
MetricDeclarationRecord/Dashboard seed constructors, PromqlQuery::parse,
LogStreamHeader (detect/split_row/column_pattern/message_pattern), and
api.rs's EngineHandle::ask/*Error::into_response/ParamDeclarationDto::reserved
all had a single obvious receiver and are now methods on it instead of
free functions taking that type as a parameter.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
Free functions that took the engine's index/sort-helper-index state as
a parameter now live on EngineSnapshot (or, for a handful of clean
single-receiver cases, on MetricPrimitive/CompareOp/CompiledFilter/
EngineQuery). Route handlers in api.rs and alert evaluation in
alerts.rs are updated to call the new methods; alerts.rs now threads
an &EngineSnapshot through instead of a raw &BTreeMap so the per-line
alert backfill path doesn't clone the whole index just to call a method.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
Centralizes engine tuning values, poll intervals, logfmt column
regex fragments, and app/settings identity strings that were
previously defined ad hoc across engine.rs, ingest.rs, logfmt.rs,
profile.rs, settings.rs, and main.rs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
- 076: gate auto-declared log-stream column/message params
(`ParamDeclaration::log_stream_gated`) behind `is_log_stream_line` at
extraction time, since their fixed-offset regex has no content anchor
and previously extracted garbage from continuation lines and the
header row itself (the regex crate doesn't support lookaround, so
this couldn't be baked into the regex text).
- 074: end-to-end regression test confirms sorting a real `log stream`
capture by a log-stream column (e.g. PID) now actually reorders
results, since 076's fix removes the garbage values that were
corrupting the sort helper index; also add a "Building sort
index…" indicator in the timeline UI for a slow first-time sort.
- 072: attach the reserved `ottyl_fmt` pseudo-param to each
`LogEntryDto`'s params so it can be picked as a Log Preview column
(previously listed as pickable but always blank).
- 073: track a rolling 5s high-watermark of ingestion-to-engine-processing
lag and surface it in the nav stats popover next to approx_bytes_total.
- 071: group the regex(...) help button back with the filter bar's other
controls, and restructure the Filtering docs section into scannable
per-feature subsections (also fixes stale log_type/process/pid mention).
Tickets 070 (CLI scoping) and 075 (API unresponsiveness investigation)
are left in place per their own scope — both are explicitly
conversation/investigation tasks, not implementation tickets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
Adds a sessions/<name>.json registry (068) so a running ottyl instance can
be discovered by name; an `ottyl mcp --session <name>` subcommand (067)
that starts a short-lived MCP server over stdio (via the official rmcp
SDK), proxying query_logs/declare_param/declare_metric/create_dashboard/
add_panel/get_dashboard tool calls onto the existing HTTP API; and a
SKILL.md (069), embedded into the binary and served at GET /SKILL.md, that
documents how an agent connects to and drives a running instance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
- 059: log stream header/column detection is now generic
whitespace-columnar splitting (2+ spaces, single tab) instead of one
fixed six-column regex. Declared params come from whatever columns
the header actually names, plus a trailing unlabelled message field
that's never swallowed by the last named column.
- 060: timeline sort-by-property now walks an incrementally-maintained
per-sort-key helper index (BTreeMap keyed so "has the param, ordered
by value" sorts before "missing it, ordered by ottyl_seq") instead of
materializing and re-sorting the full filtered range on every query.
- 061: helper indexes are swept for staleness on ingest and dropped
after 5 minutes unused; a purged (or never-built) index transparently
rebuilds via the same lazy one-time scan on the next sorted query.
- Also invalidate a param's helper index on manual re-declaration,
removal, or ClearParams, so a redefined regex can't leave a sort
helper index serving values derived under the old declaration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
- 058: poll loop now refetches param declarations so auto-declared
params appear live without a page reload
- 062: browser tab title says "ottyl" instead of "webapp"
- 063: declared params list omits the redundant capture-group suffix
when it matches the param name
- 064: dashboard autosave no longer clobbers in-flight PromQL edits;
reconciliation is guarded against edits made after the save request
was sent
- 065: filter bar placeholder/help icons lead with filter-bar syntax
instead of regex
- 066: filter bar surfaces the active time-range preset/sort order and
resyncs its draft when the value prop changes externally
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
A rate() series over a metric whose occurrences are confined to a short
burst within a much wider queried range (e.g. sum by (code)
(rate(KeyPressed[5m])) against a ~10h log where all key presses happen
in the first 10s) rendered as broken: the line simply stopped at the
last real occurrence, and the chart's x-axis auto-scaled to the tiny
span of actual data points rather than the queried range, silently
narrowing a 10-hour view down to ~10 seconds with no indication anything
was off.
Backend (src/engine.rs): windowed_rate_series now appends a synthetic
boundary point at start_ms/end_ms (computed via the existing
rate::rate_at against the original occurrence points, so a boundary
inside a still-warm window gets its true partial value rather than a
hardcoded 0) whenever the real series doesn't already reach that edge.
Wired into both the bare rate() and sum()/avg(rate()) evaluate_promql
branches, which already had start_ms/end_ms in scope.
Frontend (PanelChart.tsx/Dashboards.tsx): the line chart's XAxis now
uses an explicit numeric domain pinned to the panel's queried range
(passed down from Dashboards.tsx's PanelPreview) instead of recharts'
default auto-scale-to-data-extent, so sparse data can no longer make
the axis look narrower than what was actually queried.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
GET /api/params now includes the three non-deletable ottyl_* reserved
entries (ticket 055), so the "clear removes everything" assertion needs
to count only real (non-reserved) declarations.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Resolved conflicts with ticket 051's unknown-metric guard in evaluate_promql
(reordered so ottyl_fmt's categorical rejection and the reserved-name
skip both run before 051's declaration/bucket-data check, since
ottyl_seq/ottyl_ts are synthesized from the log index and never have a
MetricDeclaration or bucket entries) and merged both tickets' additions
to the reserved-params Docs.tsx paragraph.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
auto_declare_params (src/engine.rs) previously never overwrote or pruned a
declaration, so a burst of unusual key=value-shaped lines could permanently
inflate the params UI forever. A real macOS `log stream` dump
(huig_dump.txt, copied into src/fixtures/log_stream_sample.txt) discovered
220 distinct auto-declared params from under 700 lines, mostly one-off
WiFi/radio stat counters (20+ pairs on a single line), plus base64-blob
false positives from key()'s greedy alnum matching landing right before
`==` padding.
Fix, combining two bounds per the ticket's options:
- src/logfmt.rs: extract_pairs/pair now rejects a candidate pair whose
bareword value starts with a bare '=' (looks_like_base64_padding_artifact)
— the exact shape produced when a base64/hash blob's padding sits right
after a run of alnum chars that key() consumed as if it were a key.
- src/engine.rs: auto_declare_params now enforces a per-line cap
(MAX_NEW_AUTO_PARAMS_PER_LINE = 8) on new params from one line, and a
total ceiling (MAX_AUTO_DECLARED_PARAMS = 64) on auto-derived params
overall — the per-line cap alone doesn't bound unbounded key *variety*
across many lines, so both are needed.
Also adds structured extraction for log stream's fixed-column shape
(Timestamp Thread Type Activity PID TTL, then `process: message`):
detect_format gains a new "log_stream" ottyl_fmt value (checked first,
since it's the most specific of the four), and
auto_declare_log_stream_params declares log_type/process/pid as standalone
queryable params the first time such a line is seen — derived positionally
from the column layout rather than =-scanning, so there's no ambiguity or
false-positive risk. This is a separate always-attempted extraction
alongside the existing key=value pass, not a replacement for it: a log
stream line's message tail can still contain real key=value pairs (e.g.
the WiFi stat lines) that auto_declare_params keeps discovering
independently. This also fully addresses ticket 054, confirming huig_dump.txt
is space-padded fixed-width columns, not TSV, and that no separate TSV
format is needed.
Throughput (src/fixtures/log_stream_sample.txt x50, release build):
before (caps disabled) ~369k lines/sec, 206 auto-declared params;
after (caps enabled) ~1.20M lines/sec, 64 auto-declared params — about
3.2x faster, confirming the ever-growing declaration set was the
bottleneck rather than per-line regex/nom scanning cost.
Removes tickets/050 and tickets/054 now that both are implemented.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
evaluate_promql previously resolved every metric name through bucket_values,
which only returns data for a real MetricDeclaration — so a dashboard panel
querying ottyl_seq/ottyl_ts/ottyl_fmt (already supported in the filter bar
and alert conditions) silently returned an empty series. ottyl_seq/ottyl_ts
are now synthesized directly from the ingested-log index instead of requiring
a declaration; ottyl_fmt is categorical (json/logfmt/none) with no sensible
numeric reading, so it's rejected up front with a clear error pointing back
at the filter bar rather than returning nothing.
GET /api/params now also lists the three reserved names (marked `reserved:
true`, distinct from `auto_derived`) purely for discoverability — merged in
at the API layer only, not the engine's internal ParamDeclarations query, so
profile-switch removal and metric-param-promotion logic are unaffected.
declare_param also now rejects declaring a param under one of these names.
Docs.tsx and the Params panel are updated to reflect the new dashboard
support and the reserved/non-editable listing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Wires the engine's existing RemoveMetricDeclaration message (previously
only used internally by delete_profile's cleanup) up to a new DELETE
route, persists the removal into the active profile so it survives a
restart, and adds a delete affordance in the Metrics UI.
Decisions:
- Bucket-value history under the deleted metric's name is purged, not
left orphaned (engine.rs's new remove_metric_declaration reuses
update_metric's clear_metric_data helper). This is what lets a
dashboard panel or PromQL query still referencing the metric by name
fail with a readable "unknown metric" error instead of silently
returning stale data.
- evaluate_promql now errors when a metric has neither a declaration nor
any bucket data, since a metric ingested via --metrics-url scraping
never gets a MetricDeclaration but must stay queryable. Verified with
a seeded fixture (promql_query_against_deleted_metric_fails_with_readable_error
in src/api.rs).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
DELETE /api/params clears the entire standalone-param set — manual and
auto-derived alike — then immediately re-runs auto_declare_params against
every IndexKey::IngestedLog entry still held in the engine's index (oldest
to newest), so params come back fresh from currently-retained log text
without requiring new lines to be ingested. This is a deliberate full
reset, unlike reload_engine_declarations's profile-switch cleanup, which
preserves auto-derived params since those are engine-discovered facts
rather than profile state; here the immediate re-extraction step makes
clearing them safe.
Settings::clear_param_declarations empties the active profile's persisted
params so they don't reappear on restart. The webapp's Params section gets
a single "Clear all" button (with a confirm dialog and a status line
reporting removed/re-derived/logs_considered) rather than a per-param
delete loop; logs_considered surfaces that lines already evicted by
max_logs can't regain their params.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Extends ticket 048's light_green pattern (light-inverted surface/border/text
scale + light's neutral danger colours, same base and accent hues as the
dark-surfaced counterpart) to the remaining five colour schemes across all
six files that must stay in sync: index.css, themes.ts, api.ts, settings.rs,
useSettingsPanel.ts, and NavMenu.tsx.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
The "Sort by" / "fall back to ottyl_seq" control now lives in
LogTimeline's header next to the time-range presets, since the setting
actually drives the shared range/log fetch rather than being a
Log-Preview-only concern. LogSortSettings and its load/save helpers
moved into LogTimeline.tsx alongside SortPicker; the localStorage key
was renamed to ottyl.logTimeline.sort, with loadLogSortSettings falling
back to the old ottyl.logPreview.sort key so an existing choice isn't
lost on first load after the move. LogPreview now just renders logs in
whatever order the caller already sorted them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
There is no DELETE route, no persisted-removal path, and no UI affordance
for removing a metric once declared — the engine's RemoveMetricDeclaration
message only ever fires as part of whole-profile deletion today.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
The settings panel had two separate sections — "Colour scheme" (built-in
palettes) and "CSS" (imported/linked stylesheets) — even though the data
model (tickets 012/020) already treats both as equally-selectable themes
via custom_theme_id. Merge them into one "Style" section presenting a
single list: built-in scheme pills plus imported/linked entries, any of
which can be a palette swap or a full layout override, with copy that no
longer implies "colour scheme" and "CSS" are separate features.
Traced and manually verified (via a running dev server) that a
user-supplied stylesheet is not constrained to --c-* variable overrides:
import_css/get_css_content in src/api.rs and src/settings.rs store and
serve the raw content unmodified (no sanitization or selector scoping),
and applyCustomThemeCss injects it as a plain unscoped <link> tag — a
stylesheet with e.g. `flex-direction`/`font-size` rules on real selectors
applies exactly as authored, alongside the required --c-* declarations.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Every built-in theme now exposes three accents (--c-accent/-2/-3) instead
of one, so PanelChart's series palette draws distinct colours per theme
via color-mix() rather than a fixed, theme-independent hex list. Existing
presets keep their current base surface/border/text/accent look — the two
new accent vars are purely additive. Adds a new "Light Green" scheme
(light-surfaced, same green/magenta hue family as "Green") selectable
alongside the existing built-ins, both in ColorScheme (frontend + backend)
and the settings picker.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
worktree-agent-a19b9ef959dcda6b2 (ticket 049) was branched from before "Auto
derived format" (1a481d3) landed on main, so its synchronous
persist_metric_declaration(_update) and ProfileStore::record_metric_declaration(_update)
didn't know about promoting auto-derived engine params into the persisted
profile. This merge reconciles the two:
- src/api.rs: kept the async persist_metric_declaration/_update methods and
promoted_param_records helper from "Auto derived format", but switched their
persistence from ProfileStore::load/record_metric_declaration to the new
Settings-based settings.record_metric_declaration(_update), matching ticket
049's single-settings.json model. Dropped the now-unused
active_profile_location helper in favor of locking self.settings directly.
- src/profile.rs: removed the standalone ProfileStore struct/impl entirely
(049 folds profiles into Settings), keeping load_legacy_profile_store for
migrating a pre-049 profiles.json. Extended record_metric_declaration and
record_metric_declaration_update (now methods on Settings) to keep taking
promoted_params: &[ParamDeclarationRecord], persisting promoted auto-derived
params alongside the metric via Settings::save. Moved the ticket 037/038
legacy-shape migration (migrate_legacy_metric_params /
migrate_duration_value_params) out of the removed ProfileStore::load into a
new pub(crate) migrate_legacy_profile_shapes(profiles), invoked from
Settings::load (via a new Settings::migrate_legacy_profile_shapes) so
profiles inlined in settings.json still get migrated forward on every load.
Rewrote the ProfileStore-based tests (legacy-shape migration tests, the
no-op-migration test, and the auto-derived-param promotion round-trip test)
to exercise Settings directly, including a write_legacy_settings_json helper
that replaces the old write_legacy_profiles_json.
- src/api.rs tests: rewrote declaring_metric_against_auto_derived_param_promotes_it_into_the_profile
to create/activate the profile via settings_handle.lock() and reload via
Settings::load instead of ProfileStore.
Also fixed one clippy::collapsible_if warning introduced by the new
Settings::migrate_legacy_profile_shapes.
cargo build, cargo test (222 passed, 0 failed, 0 ignored), and
cargo clippy --all-targets all pass; no new clippy warnings beyond
pre-existing ones in files untouched by this merge.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
/api/logs now accepts sort_by (an already-declared param name) and
sort_missing_fallback: lines carrying that param are ordered by its value,
while lines missing it are either appended in ottyl_seq order (fallback,
the default) or dropped from the response. Default ottyl_ts-tracking
order is unchanged when sort_by is omitted.
The log viewer (both the inline panel and its popout) exposes this as a
"Sort by" control next to the column picker, persisted to localStorage
under one shared key (ottyl.logPreview.sort) — the same global-viewer-
setting convention already used for visible param columns and the log
viewer's height, since there's no per-dashboard scope for this to divide
by.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Fold ProfileStore's separate profiles.json into Settings itself, so
settings.json is the single source of truth instead of two files kept
in sync by hand. Settings now carries `profiles: Vec<Profile>` and
`active_profile: Option<String>` directly; every former ProfileStore
mutation (create, delete, set_active, record_param_declaration,
record_metric_declaration, dashboard CRUD, ensure_default) is now a
method on Settings that persists via Settings::save.
A custom Deserialize (via SettingsOnDisk) keeps an old settings.json
(with the removed Profiles pointer struct) loading cleanly, and
Settings::load runs a one-time migration: if profiles is empty and a
legacy <settings_dir>/profiles/profiles.json exists, its contents are
merged in, settings.json is resaved, and the old file is renamed so it
isn't re-imported.
Updated all call sites in api.rs and main.rs to the single-path API,
and ported profile.rs's ProfileStore tests to exercise Settings
directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
Ticket 044: `sequence` and `timestamp` collided with ordinary user-declared
param names and didn't signal they're ottyl-owned, so they're renamed
(breaking change, no back-compat alias) to `ottyl_seq`/`ottyl_ts` in the
filter grammar (src/filter_parser.pest, src/filter_parser.rs), alert query
restriction/evaluation (src/alerts.rs), and the filter-bar docs/placeholder
in the webapp.
Also adds a new reserved param `ottyl_fmt` exposing which format a line was
auto-detected as ("json"/"logfmt"/"none") via a new
`crate::logfmt::detect_format`, wired into the same filter grammar and alert
restriction list as `ottyl_seq`/`ottyl_ts`. A mixed-format fixture test in
engine.rs verifies `ottyl_fmt` reflects the actual detected format per line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
