phughk/ottyl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Compare commits

base: phughk/ottyl:v0.1.0
Branches Tags
phughk/ottyl:main
phughk/ottyl:v0.1.0
...
compare: phughk/ottyl:65330d37e0cb0b4a0ceb970e77df9d8ebbf38c09
Branches Tags
phughk/ottyl:main
phughk/ottyl:v0.1.0
75 Commits
v0.1.0 ... 65330d37e0
Author SHA1 Message Date
Hugh Kaznowski 65330d37e0 Add tickets, ideas and samply ignore 2026-09-06 20:45:33 +01:00
Hugh Kaznowski d44c14a83b Improve ingestion performance 2026-09-06 19:02:44 +01:00
Hugh KaznowskiandClaude Sonnet 5 f19fa50a25 Split src/engine/mod.rs into handle/message/snapshot submodules 
Pure code-motion refactor, no logic changes. Follows the repo convention
of one struct/enum + its impl per module file:

- engine/handle.rs: the left-right write/read plumbing (EngineOp,
  IndexWriter, IndexReadHandle, IndexReader, EngineHandle) and the actor
  loop (spawn/run) that owns EngineSnapshot.
- engine/message.rs: EngineQuery/EngineMessage/EngineResponse.
- engine/snapshot/: EngineSnapshot itself, split by concern into
  params.rs (declaration + auto-detect), metrics.rs (declare/apply/
  backfill), filter.rs (search-bar filter compilation/eval), sort.rs
  (sort-helper indexes), and query.rs (handle_query_inner + PromQL
  evaluation), with mod.rs holding the struct and the accessors shared
  across all of them.
- engine/tests.rs: the inline test module, kept as one file (per the
  task's own escape hatch) since most tests exercise ingest -> declare
  -> query across several of the new submodules at once rather than one
  in isolation.

Visibility was narrowed per call site: pub(super) where only sibling
snapshot submodules need an item, pub(crate) where engine::handle or
engine::tests (outside the snapshot subtree) also need it.

cargo build --all-targets and cargo test are green (291 passed, 1
ignored, same as before the split).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 18:36:53 +01:00
Hugh Kaznowski 3b3ace146a Increase mcp scope 2026-09-06 17:34:36 +01:00
Hugh KaznowskiandClaude Sonnet 5 a810ecf396 Split src/ into an api/cli/engine/config module hierarchy 
- src/api/: HTTP router + DTOs + the MCP server (the HTTP/MCP-facing
  surface): mod.rs (formerly api.rs), dto.rs, mcp_server.rs.
- src/cli/: argument parsing and process entry (formerly inline in
  main.rs) plus session registry: mod.rs, sessions.rs. main.rs is now
  a thin entry point that just declares modules and calls cli::run().
- src/engine/: the actor + everything it depends on to parse/index/
  query log lines: mod.rs (formerly engine.rs), alerts.rs, promql.rs,
  rate.rs, ingest.rs, logfmt.rs, filter_parser.rs/.pest,
  filter_datetime.rs, index_key.rs, data_value.rs, metric_primitive.rs,
  timestamp.rs.
- src/config/: persisted configuration: mod.rs (formerly settings.rs),
  profile.rs.
- src/consts.rs stays at the crate root, used from every module above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:15:39 +01:00
Hugh KaznowskiandClaude Sonnet 5 311713986a Wrap the left-right index in IndexWriter/IndexReader types 
The engine actor and API handlers previously called left_right's
WriteHandle/ReadHandleFactory methods (append/publish/enter/handle)
directly. IndexWriter now exposes mutate()/peek() and IndexReader
exposes handle() (returning an IndexReadHandle with its own enter()),
so how the single-writer/multi-reader index is accessed is named and
scoped to these two types instead of being raw left_right calls
scattered through engine.rs/api.rs/profile.rs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:10:23 +01:00
Hugh KaznowskiandClaude Sonnet 5 3c2995acc1 Convert remaining clean-receiver free functions to methods 
Profile (legacy-shape migration, load_into_engine, load_legacy_store),
MetricDeclarationRecord/Dashboard seed constructors, PromqlQuery::parse,
LogStreamHeader (detect/split_row/column_pattern/message_pattern), and
api.rs's EngineHandle::ask/*Error::into_response/ParamDeclarationDto::reserved
all had a single obvious receiver and are now methods on it instead of
free functions taking that type as a parameter.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:08:29 +01:00
Hugh KaznowskiandClaude Sonnet 5 529f79e9da Convert engine.rs index-manipulating functions into EngineSnapshot methods 
Free functions that took the engine's index/sort-helper-index state as
a parameter now live on EngineSnapshot (or, for a handful of clean
single-receiver cases, on MetricPrimitive/CompareOp/CompiledFilter/
EngineQuery). Route handlers in api.rs and alert evaluation in
alerts.rs are updated to call the new methods; alerts.rs now threads
an &EngineSnapshot through instead of a raw &BTreeMap so the per-line
alert backfill path doesn't clone the whole index just to call a method.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:03:19 +01:00
Hugh KaznowskiandClaude Sonnet 5 b6401c1c3c Strip all comments outside src/consts.rs 
Removes doc comments, module comments, and inline comments across
the backend so the only comments left in the codebase are the brief
ones on the constants they document.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 16:34:42 +01:00
Hugh KaznowskiandClaude Sonnet 5 284e3562a2 Move scattered magic constants into src/consts.rs 
Centralizes engine tuning values, poll intervals, logfmt column
regex fragments, and app/settings identity strings that were
previously defined ad hoc across engine.rs, ingest.rs, logfmt.rs,
profile.rs, settings.rs, and main.rs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 16:29:03 +01:00
Hugh Kaznowski 1c635bc530 Left-right single-writer, multi-reader index 2026-09-06 16:05:04 +01:00
Hugh KaznowskiandClaude Sonnet 5 9fc7540a5b Resolve tickets 071-074, 076: log-stream contamination, sort, ottyl_fmt column, ingest lag 
- 076: gate auto-declared log-stream column/message params
  (`ParamDeclaration::log_stream_gated`) behind `is_log_stream_line` at
  extraction time, since their fixed-offset regex has no content anchor
  and previously extracted garbage from continuation lines and the
  header row itself (the regex crate doesn't support lookaround, so
  this couldn't be baked into the regex text).
- 074: end-to-end regression test confirms sorting a real `log stream`
  capture by a log-stream column (e.g. PID) now actually reorders
  results, since 076's fix removes the garbage values that were
  corrupting the sort helper index; also add a "Building sort
  index…" indicator in the timeline UI for a slow first-time sort.
- 072: attach the reserved `ottyl_fmt` pseudo-param to each
  `LogEntryDto`'s params so it can be picked as a Log Preview column
  (previously listed as pickable but always blank).
- 073: track a rolling 5s high-watermark of ingestion-to-engine-processing
  lag and surface it in the nav stats popover next to approx_bytes_total.
- 071: group the regex(...) help button back with the filter bar's other
  controls, and restructure the Filtering docs section into scannable
  per-feature subsections (also fixes stale log_type/process/pid mention).

Tickets 070 (CLI scoping) and 075 (API unresponsiveness investigation)
are left in place per their own scope — both are explicitly
conversation/investigation tasks, not implementation tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 15:12:07 +01:00
Hugh Kaznowski f34dbba6c7 Update skill and tickets 2026-09-06 14:37:03 +01:00
Hugh Kaznowski fc8de1a5b0 Remove wishlist 2026-09-06 14:00:05 +01:00
Hugh KaznowskiandClaude Sonnet 5 c77404ba4e Resolve tickets 067-069: named sessions, MCP server, embedded SKILL.md 
Adds a sessions/<name>.json registry (068) so a running ottyl instance can
be discovered by name; an `ottyl mcp --session <name>` subcommand (067)
that starts a short-lived MCP server over stdio (via the official rmcp
SDK), proxying query_logs/declare_param/declare_metric/create_dashboard/
add_panel/get_dashboard tool calls onto the existing HTTP API; and a
SKILL.md (069), embedded into the binary and served at GET /SKILL.md, that
documents how an agent connects to and drives a running instance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 13:57:30 +01:00
Hugh KaznowskiandClaude Sonnet 5 de9ba6f01b Resolve tickets 059-061: generic log-stream columns + sort helper indexes 
- 059: log stream header/column detection is now generic
  whitespace-columnar splitting (2+ spaces, single tab) instead of one
  fixed six-column regex. Declared params come from whatever columns
  the header actually names, plus a trailing unlabelled message field
  that's never swallowed by the last named column.
- 060: timeline sort-by-property now walks an incrementally-maintained
  per-sort-key helper index (BTreeMap keyed so "has the param, ordered
  by value" sorts before "missing it, ordered by ottyl_seq") instead of
  materializing and re-sorting the full filtered range on every query.
- 061: helper indexes are swept for staleness on ingest and dropped
  after 5 minutes unused; a purged (or never-built) index transparently
  rebuilds via the same lazy one-time scan on the next sorted query.
- Also invalidate a param's helper index on manual re-declaration,
  removal, or ClearParams, so a redefined regex can't leave a sort
  helper index serving values derived under the old declaration.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 13:23:10 +01:00
Hugh KaznowskiandClaude Sonnet 5 4ace654019 Resolve tickets 058, 062-066: frontend UI fixes 
- 058: poll loop now refetches param declarations so auto-declared
  params appear live without a page reload
- 062: browser tab title says "ottyl" instead of "webapp"
- 063: declared params list omits the redundant capture-group suffix
  when it matches the param name
- 064: dashboard autosave no longer clobbers in-flight PromQL edits;
  reconciliation is guarded against edits made after the save request
  was sent
- 065: filter bar placeholder/help icons lead with filter-bar syntax
  instead of regex
- 066: filter bar surfaces the active time-range preset/sort order and
  resyncs its draft when the value prop changes externally

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 13:04:26 +01:00
Hugh Kaznowski a03ec1fb93 Create tickets 2026-09-06 13:00:21 +01:00
Hugh Kaznowski 4158c63086 Add nom parsers for go and python 2026-09-06 10:45:55 +01:00
Hugh Kaznowski 8f71f2d056 Icons, docs formats 2026-09-06 10:36:26 +01:00
Hugh Kaznowski 17efe222df Add icon 2026-09-06 10:16:11 +01:00
Hugh KaznowskiandClaude Sonnet 5 79d252c1df Implement ticket 057: fix rate() line charts for sparse/bursty occurrences 
A rate() series over a metric whose occurrences are confined to a short
burst within a much wider queried range (e.g. sum by (code)
(rate(KeyPressed[5m])) against a ~10h log where all key presses happen
in the first 10s) rendered as broken: the line simply stopped at the
last real occurrence, and the chart's x-axis auto-scaled to the tiny
span of actual data points rather than the queried range, silently
narrowing a 10-hour view down to ~10 seconds with no indication anything
was off.

Backend (src/engine.rs): windowed_rate_series now appends a synthetic
boundary point at start_ms/end_ms (computed via the existing
rate::rate_at against the original occurrence points, so a boundary
inside a still-warm window gets its true partial value rather than a
hardcoded 0) whenever the real series doesn't already reach that edge.
Wired into both the bare rate() and sum()/avg(rate()) evaluate_promql
branches, which already had start_ms/end_ms in scope.

Frontend (PanelChart.tsx/Dashboards.tsx): the line chart's XAxis now
uses an explicit numeric domain pinned to the panel's queried range
(passed down from Dashboards.tsx's PanelPreview) instead of recharts'
default auto-scale-to-data-extent, so sparse data can no longer make
the axis look narrower than what was actually queried.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:53:12 +01:00
Hugh KaznowskiandClaude Sonnet 5 5493358db8 Fix ticket 052's clear-params test for ticket 055's reserved params 
GET /api/params now includes the three non-deletable ottyl_* reserved
entries (ticket 055), so the "clear removes everything" assertion needs
to count only real (non-reserved) declarations.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:49:02 +01:00
Hugh KaznowskiandClaude Sonnet 5 ec98a3ab18 Merge ticket 055: make ottyl_* reserved params usable in dashboard PromQL queries 
Resolved conflicts with ticket 051's unknown-metric guard in evaluate_promql
(reordered so ottyl_fmt's categorical rejection and the reserved-name
skip both run before 051's declaration/bucket-data check, since
ottyl_seq/ottyl_ts are synthesized from the log index and never have a
MetricDeclaration or bucket entries) and merged both tickets' additions
to the reserved-params Docs.tsx paragraph.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:44:52 +01:00
Hugh KaznowskiandClaude Sonnet 5 85f2a5c0ac Merge ticket 052: allow clearing all params and re-extracting from ingested logs 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:44:09 +01:00
Hugh KaznowskiandClaude Sonnet 5 30c301d595 Merge ticket 051: allow deleting a declared metric 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:44:05 +01:00
Hugh KaznowskiandClaude Sonnet 5 fdad154c1a Merge ticket 050/054: bound auto-declared param cardinality, recognize log stream format 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:44:01 +01:00
Hugh KaznowskiandClaude Sonnet 5 6cdb9dda2c Merge ticket 056: add Light variants for remaining colour schemes 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:43:57 +01:00
Hugh KaznowskiandClaude Sonnet 5 914aaf88b3 Merge ticket 053: move sort selector from Log Preview to Timeline 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:43:52 +01:00
Hugh KaznowskiandClaude Sonnet 5 b15fbcee04 Add ticket 057: rate() line chart x-axis collapses to sparse occurrence window 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:43:48 +01:00
Hugh KaznowskiandClaude Sonnet 5 f48d3c6ffb Bound auto-declared param cardinality and recognize log stream format (ticket 050/054) 
auto_declare_params (src/engine.rs) previously never overwrote or pruned a
declaration, so a burst of unusual key=value-shaped lines could permanently
inflate the params UI forever. A real macOS `log stream` dump
(huig_dump.txt, copied into src/fixtures/log_stream_sample.txt) discovered
220 distinct auto-declared params from under 700 lines, mostly one-off
WiFi/radio stat counters (20+ pairs on a single line), plus base64-blob
false positives from key()'s greedy alnum matching landing right before
`==` padding.

Fix, combining two bounds per the ticket's options:
- src/logfmt.rs: extract_pairs/pair now rejects a candidate pair whose
  bareword value starts with a bare '=' (looks_like_base64_padding_artifact)
  — the exact shape produced when a base64/hash blob's padding sits right
  after a run of alnum chars that key() consumed as if it were a key.
- src/engine.rs: auto_declare_params now enforces a per-line cap
  (MAX_NEW_AUTO_PARAMS_PER_LINE = 8) on new params from one line, and a
  total ceiling (MAX_AUTO_DECLARED_PARAMS = 64) on auto-derived params
  overall — the per-line cap alone doesn't bound unbounded key *variety*
  across many lines, so both are needed.

Also adds structured extraction for log stream's fixed-column shape
(Timestamp Thread Type Activity PID TTL, then `process: message`):
detect_format gains a new "log_stream" ottyl_fmt value (checked first,
since it's the most specific of the four), and
auto_declare_log_stream_params declares log_type/process/pid as standalone
queryable params the first time such a line is seen — derived positionally
from the column layout rather than =-scanning, so there's no ambiguity or
false-positive risk. This is a separate always-attempted extraction
alongside the existing key=value pass, not a replacement for it: a log
stream line's message tail can still contain real key=value pairs (e.g.
the WiFi stat lines) that auto_declare_params keeps discovering
independently. This also fully addresses ticket 054, confirming huig_dump.txt
is space-padded fixed-width columns, not TSV, and that no separate TSV
format is needed.

Throughput (src/fixtures/log_stream_sample.txt x50, release build):
before (caps disabled) ~369k lines/sec, 206 auto-declared params;
after (caps enabled) ~1.20M lines/sec, 64 auto-declared params — about
3.2x faster, confirming the ever-growing declaration set was the
bottleneck rather than per-line regex/nom scanning cost.

Removes tickets/050 and tickets/054 now that both are implemented.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:42:50 +01:00
Hugh KaznowskiandClaude Sonnet 5 ff56558b2e Make ottyl_seq/ottyl_ts usable in dashboard PromQL queries (ticket 055) 
evaluate_promql previously resolved every metric name through bucket_values,
which only returns data for a real MetricDeclaration — so a dashboard panel
querying ottyl_seq/ottyl_ts/ottyl_fmt (already supported in the filter bar
and alert conditions) silently returned an empty series. ottyl_seq/ottyl_ts
are now synthesized directly from the ingested-log index instead of requiring
a declaration; ottyl_fmt is categorical (json/logfmt/none) with no sensible
numeric reading, so it's rejected up front with a clear error pointing back
at the filter bar rather than returning nothing.

GET /api/params now also lists the three reserved names (marked `reserved:
true`, distinct from `auto_derived`) purely for discoverability — merged in
at the API layer only, not the engine's internal ParamDeclarations query, so
profile-switch removal and metric-param-promotion logic are unaffected.
declare_param also now rejects declaring a param under one of these names.

Docs.tsx and the Params panel are updated to reflect the new dashboard
support and the reserved/non-editable listing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:42:19 +01:00
Hugh KaznowskiandClaude Sonnet 5 466c81fa94 Add DELETE /api/metrics/{name} to allow deleting a declared metric (ticket 051) 
Wires the engine's existing RemoveMetricDeclaration message (previously
only used internally by delete_profile's cleanup) up to a new DELETE
route, persists the removal into the active profile so it survives a
restart, and adds a delete affordance in the Metrics UI.

Decisions:
- Bucket-value history under the deleted metric's name is purged, not
  left orphaned (engine.rs's new remove_metric_declaration reuses
  update_metric's clear_metric_data helper). This is what lets a
  dashboard panel or PromQL query still referencing the metric by name
  fail with a readable "unknown metric" error instead of silently
  returning stale data.
- evaluate_promql now errors when a metric has neither a declaration nor
  any bucket data, since a metric ingested via --metrics-url scraping
  never gets a MetricDeclaration but must stay queryable. Verified with
  a seeded fixture (promql_query_against_deleted_metric_fails_with_readable_error
  in src/api.rs).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:41:07 +01:00
Hugh KaznowskiandClaude Sonnet 5 4e37e2a7ee Add bulk clear-params action that re-extracts from retained logs (ticket 052) 
DELETE /api/params clears the entire standalone-param set — manual and
auto-derived alike — then immediately re-runs auto_declare_params against
every IndexKey::IngestedLog entry still held in the engine's index (oldest
to newest), so params come back fresh from currently-retained log text
without requiring new lines to be ingested. This is a deliberate full
reset, unlike reload_engine_declarations's profile-switch cleanup, which
preserves auto-derived params since those are engine-discovered facts
rather than profile state; here the immediate re-extraction step makes
clearing them safe.

Settings::clear_param_declarations empties the active profile's persisted
params so they don't reappear on restart. The webapp's Params section gets
a single "Clear all" button (with a confirm dialog and a status line
reporting removed/re-derived/logs_considered) rather than a per-param
delete loop; logs_considered surfaces that lines already evicted by
max_logs can't regain their params.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:39:22 +01:00
Hugh KaznowskiandClaude Sonnet 5 d59b3aec38 Add light variants for red, blue, yellow, pink, orange colour schemes 
Extends ticket 048's light_green pattern (light-inverted surface/border/text
scale + light's neutral danger colours, same base and accent hues as the
dark-surfaced counterpart) to the remaining five colour schemes across all
six files that must stay in sync: index.css, themes.ts, api.ts, settings.rs,
useSettingsPanel.ts, and NavMenu.tsx.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:37:21 +01:00
Hugh KaznowskiandClaude Sonnet 5 78a6672ced Move log sort selector from Log Preview to Timeline header 
The "Sort by" / "fall back to ottyl_seq" control now lives in
LogTimeline's header next to the time-range presets, since the setting
actually drives the shared range/log fetch rather than being a
Log-Preview-only concern. LogSortSettings and its load/save helpers
moved into LogTimeline.tsx alongside SortPicker; the localStorage key
was renamed to ottyl.logTimeline.sort, with loadLogSortSettings falling
back to the old ottyl.logPreview.sort key so an existing choice isn't
lost on first load after the move. LogPreview now just renders logs in
whatever order the caller already sorted them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:34:46 +01:00
Hugh Kaznowski d17cfe9b83 Update claude and tickets 2026-09-06 09:30:38 +01:00
Hugh KaznowskiandClaude Sonnet 5 2908cccd76 Add ticket 051: allow deleting a declared metric 
There is no DELETE route, no persisted-removal path, and no UI affordance
for removing a metric once declared — the engine's RemoveMetricDeclaration
message only ever fires as part of whole-profile deletion today.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 09:11:31 +01:00
Hugh KaznowskiandClaude Sonnet 5 5cb2dfde13 Remove ticket 047 (implemented: unified style presets, colour + layout as one concept) 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:17:39 +01:00
Hugh Kaznowski 8ab5edc96b Merge ticket 047: unified style presets (colour + layout as one concept) 2026-09-06 02:17:27 +01:00
Hugh KaznowskiandClaude Sonnet 5 5b31f51526 Unify colour scheme and CSS as one Style list in settings UI (ticket 047) 
The settings panel had two separate sections — "Colour scheme" (built-in
palettes) and "CSS" (imported/linked stylesheets) — even though the data
model (tickets 012/020) already treats both as equally-selectable themes
via custom_theme_id. Merge them into one "Style" section presenting a
single list: built-in scheme pills plus imported/linked entries, any of
which can be a palette swap or a full layout override, with copy that no
longer implies "colour scheme" and "CSS" are separate features.

Traced and manually verified (via a running dev server) that a
user-supplied stylesheet is not constrained to --c-* variable overrides:
import_css/get_css_content in src/api.rs and src/settings.rs store and
serve the raw content unmodified (no sanitization or selector scoping),
and applyCustomThemeCss injects it as a plain unscoped <link> tag — a
stylesheet with e.g. `flex-direction`/`font-size` rules on real selectors
applies exactly as authored, alongside the required --c-* declarations.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:17:10 +01:00
Hugh KaznowskiandClaude Sonnet 5 7c331e76b1 Remove ticket 048 (implemented: multi-accent and brighter theme variants) 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:12:04 +01:00
Hugh Kaznowski 8e7973e4fd Merge ticket 048: multi-accent and brighter theme variants 2026-09-06 02:11:49 +01:00
Hugh KaznowskiandClaude Sonnet 5 5371d3e283 Add multi-accent palettes and a brighter Light Green theme (ticket 048) 
Every built-in theme now exposes three accents (--c-accent/-2/-3) instead
of one, so PanelChart's series palette draws distinct colours per theme
via color-mix() rather than a fixed, theme-independent hex list. Existing
presets keep their current base surface/border/text/accent look — the two
new accent vars are purely additive. Adds a new "Light Green" scheme
(light-surfaced, same green/magenta hue family as "Green") selectable
alongside the existing built-ins, both in ColorScheme (frontend + backend)
and the settings picker.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:11:33 +01:00
Hugh KaznowskiandClaude Sonnet 5 19044753c6 Remove ticket 045 (implemented: timeline/log ordering by declared param) 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:04:09 +01:00
Hugh Kaznowski 0b6442a348 Merge ticket 045: timeline/log ordering by declared param 2026-09-06 02:03:53 +01:00
Hugh KaznowskiandClaude Sonnet 5 7efaaa12f1 Remove ticket 049 (implemented: profiles stored inline in settings.json) 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:03:49 +01:00
Hugh KaznowskiandClaude Sonnet 5 7a38d008fd Merge ticket 049 (inline profiles into settings.json) with auto-derived-param promotion 
worktree-agent-a19b9ef959dcda6b2 (ticket 049) was branched from before "Auto
derived format" (1a481d3) landed on main, so its synchronous
persist_metric_declaration(_update) and ProfileStore::record_metric_declaration(_update)
didn't know about promoting auto-derived engine params into the persisted
profile. This merge reconciles the two:

- src/api.rs: kept the async persist_metric_declaration/_update methods and
  promoted_param_records helper from "Auto derived format", but switched their
  persistence from ProfileStore::load/record_metric_declaration to the new
  Settings-based settings.record_metric_declaration(_update), matching ticket
  049's single-settings.json model. Dropped the now-unused
  active_profile_location helper in favor of locking self.settings directly.
- src/profile.rs: removed the standalone ProfileStore struct/impl entirely
  (049 folds profiles into Settings), keeping load_legacy_profile_store for
  migrating a pre-049 profiles.json. Extended record_metric_declaration and
  record_metric_declaration_update (now methods on Settings) to keep taking
  promoted_params: &[ParamDeclarationRecord], persisting promoted auto-derived
  params alongside the metric via Settings::save. Moved the ticket 037/038
  legacy-shape migration (migrate_legacy_metric_params /
  migrate_duration_value_params) out of the removed ProfileStore::load into a
  new pub(crate) migrate_legacy_profile_shapes(profiles), invoked from
  Settings::load (via a new Settings::migrate_legacy_profile_shapes) so
  profiles inlined in settings.json still get migrated forward on every load.
  Rewrote the ProfileStore-based tests (legacy-shape migration tests, the
  no-op-migration test, and the auto-derived-param promotion round-trip test)
  to exercise Settings directly, including a write_legacy_settings_json helper
  that replaces the old write_legacy_profiles_json.
- src/api.rs tests: rewrote declaring_metric_against_auto_derived_param_promotes_it_into_the_profile
  to create/activate the profile via settings_handle.lock() and reload via
  Settings::load instead of ProfileStore.

Also fixed one clippy::collapsible_if warning introduced by the new
Settings::migrate_legacy_profile_shapes.

cargo build, cargo test (222 passed, 0 failed, 0 ignored), and
cargo clippy --all-targets all pass; no new clippy warnings beyond
pre-existing ones in files untouched by this merge.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 02:03:22 +01:00
Hugh KaznowskiandClaude Sonnet 5 3d2652e95b Add sort_by ordering for the log viewer, with a missing-param fallback/drop toggle (ticket 045) 
/api/logs now accepts sort_by (an already-declared param name) and
sort_missing_fallback: lines carrying that param are ordered by its value,
while lines missing it are either appended in ottyl_seq order (fallback,
the default) or dropped from the response. Default ottyl_ts-tracking
order is unchanged when sort_by is omitted.

The log viewer (both the inline panel and its popout) exposes this as a
"Sort by" control next to the column picker, persisted to localStorage
under one shared key (ottyl.logPreview.sort) — the same global-viewer-
setting convention already used for visible param columns and the log
viewer's height, since there's no per-dashboard scope for this to divide
by.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 01:58:30 +01:00
Hugh KaznowskiandClaude Sonnet 5 e8baf2b004 Inline profile storage into settings.json (ticket 049) 
Fold ProfileStore's separate profiles.json into Settings itself, so
settings.json is the single source of truth instead of two files kept
in sync by hand. Settings now carries `profiles: Vec<Profile>` and
`active_profile: Option<String>` directly; every former ProfileStore
mutation (create, delete, set_active, record_param_declaration,
record_metric_declaration, dashboard CRUD, ensure_default) is now a
method on Settings that persists via Settings::save.

A custom Deserialize (via SettingsOnDisk) keeps an old settings.json
(with the removed Profiles pointer struct) loading cleanly, and
Settings::load runs a one-time migration: if profiles is empty and a
legacy <settings_dir>/profiles/profiles.json exists, its contents are
merged in, settings.json is resaved, and the old file is renamed so it
isn't re-imported.

Updated all call sites in api.rs and main.rs to the single-path API,
and ported profile.rs's ProfileStore tests to exercise Settings
directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 01:54:04 +01:00
Hugh KaznowskiandClaude Sonnet 5 70d98562d1 Remove ticket 044 (implemented: ottyl_fmt param + ottyl_seq/ottyl_ts rename) 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 01:50:50 +01:00
Hugh Kaznowski a7147f47a0 Merge ticket 044: ottyl_fmt reserved param + ottyl_seq/ottyl_ts rename 2026-09-06 01:50:39 +01:00
Hugh KaznowskiandClaude Sonnet 5 534dc52e05 Rename reserved sequence/timestamp params to ottyl_seq/ottyl_ts; add ottyl_fmt 
Ticket 044: `sequence` and `timestamp` collided with ordinary user-declared
param names and didn't signal they're ottyl-owned, so they're renamed
(breaking change, no back-compat alias) to `ottyl_seq`/`ottyl_ts` in the
filter grammar (src/filter_parser.pest, src/filter_parser.rs), alert query
restriction/evaluation (src/alerts.rs), and the filter-bar docs/placeholder
in the webapp.

Also adds a new reserved param `ottyl_fmt` exposing which format a line was
auto-detected as ("json"/"logfmt"/"none") via a new
`crate::logfmt::detect_format`, wired into the same filter grammar and alert
restriction list as `ottyl_seq`/`ottyl_ts`. A mixed-format fixture test in
engine.rs verifies `ottyl_fmt` reflects the actual detected format per line.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 01:50:16 +01:00
Hugh KaznowskiandClaude Sonnet 5 c84276937a Remove ticket 046 (implemented: chart legend squashing) 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 01:44:25 +01:00
Hugh Kaznowski 287b81989c Merge ticket 046: chart legend squashing 2026-09-06 01:44:18 +01:00
Hugh KaznowskiandClaude Sonnet 5 8a8e3f60ec Squash chart bars/legend into top-N + Other for many-series panels 
PanelChart's BarChart and PieChart rendered one bar/slice and legend
row per PromqlSeries with no bound, so panels with many series wrapped
the 8-color palette (repeating colors) and produced unreadable,
unbounded legends. Add squashRows(), which sorts by value descending
and collapses series beyond COLORS.length - 1 into a single "Other"
bucket (sum of the rest), applied to both bar and pie chart data before
render. Panels at or below the threshold are unaffected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-06 01:44:05 +01:00
Hugh Kaznowski a97bd077a4 Created tickets relating to legends, reserved params, timeline ordering, css, etc 2026-09-06 01:41:09 +01:00
Hugh Kaznowski ac1efd9e53 Update wishlist and remove done ticets 2026-09-06 01:31:32 +01:00
Hugh Kaznowski 1a481d374b Auto derived format 2026-09-06 01:20:17 +01:00
Hugh KaznowskiandClaude Sonnet 5 ed9aa8eb45 Allow clicking a declared param to load it into the form for editing 
Mirrors Metrics.tsx's click-to-edit pattern: clicking a param chip
populates the declaration form, shows an "Editing" indicator, and swaps
the submit button for Save changes / Cancel. The name field is locked
while editing since declare_param overwrites in place by name — renaming
would silently orphan the original and anything referencing it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 23:32:06 +01:00
Hugh KaznowskiandClaude Sonnet 5 5e30e02a78 Make elapsed-since-last an explicit param extractor, not a Duration-kind special case (ticket 038) 
Previously a Value-role param with kind: Duration silently meant "elapsed
time since the previous same-labeled event" instead of parsed text — the
same MetricPrimitiveKind::Duration variant meant two unrelated things
depending on which list (label_params vs value_param) referenced it.

Add ParamDeclaration::elapsed_since_last, a third extraction source
alongside capture_group/json_field_path: when set, the param is computed
(via a new resolve_param_value, used uniformly for label_params and
value_param) rather than text-extracted, and declare_param validates it's
paired with kind: Duration. apply_metric_to_log no longer branches on
(role, kind) — it dispatches purely on what each resolved param declares.

Migrate existing profiles: migrate_duration_value_params converts any
Duration-kind param referenced as a metric's value_param into the new
extractor, and resolve_or_create_legacy_param (ticket 037's pre-existing
migration) now carries the same meaning forward for pre-037 inline
Value-role Duration matchers.

Params.tsx offers "Elapsed since previous event" as its own Extraction
option (hides the now-irrelevant message-match-regex/capture-group
fields and forces kind to duration); Metrics.tsx labels such params
accordingly wherever they're listed as label/value param choices.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 23:01:32 +01:00
Hugh KaznowskiandClaude Sonnet 5 375aa38ef8 Validate metric[param] filters against declared label params (ticket 039) 
compile_metric_comparison used to look up `param` by reading whatever key
happened to be present in a bucket's `labels` map at query time, so a typo
or a param never declared as a label on that metric silently matched
nothing instead of being rejected. Now it looks up the target metric's
MetricDeclaration (ticket 037's unified label_params/value_param shape) and
rejects the filter up front with an error naming the metric and its actual
declared label params when `param` isn't among them.

Metric names with no declaration at all are left unvalidated here, matching
prior behavior (bucket_values simply yields nothing for them).

Added tests: metric[nonexistent_param] is rejected with an error naming the
metric and its real label params; existing metric[real_param] filters still
match as before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:50:46 +01:00
Hugh KaznowskiandClaude Sonnet 5 db0d20643b Merge ticket 040 (unify rate/window implementation) 
Fix one test call site to unwrap the Result now returned by
evaluate_promql (introduced by ticket 041). Full suite (194 tests),
clippy, and webapp build all pass after merging all of 037, 040, 041,
042, and 043.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:46:31 +01:00
Hugh Kaznowski 322436eed7 Merge branch 'worktree-agent-a9ca45be433ffca98' 2026-09-05 22:45:59 +01:00
Hugh KaznowskiandClaude Sonnet 5 7ab840db6f Merge ticket 043 (histogram by() support), resolve doc-comment conflict 
Also fix 043's histogram-by-clause test to unwrap the Result now returned
by evaluate_promql (introduced by ticket 041).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:45:55 +01:00
Hugh KaznowskiandClaude Sonnet 5 9b84ab6895 Merge ticket 042 (sum/avg timestamp merge fix), resolve conflicts with 041 
Rebase 042's Count-aggregation and windowed-merge test helpers onto 041's
Result-based evaluate_promql signature and 037's declared-param-by-name
metric shape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:44:46 +01:00
Hugh KaznowskiandClaude Sonnet 5 60f4574d17 Fix ticket 041 merge conflict: update declared_value_kind for ticket 037's param shape 
The 041 branch was built against the pre-037 ParamRole/inline-params shape;
rebase the string-kind lookup and test helper onto the unified
ParamDeclaration/label_params/value_param model from ticket 037.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:43:11 +01:00
Hugh Kaznowski ce3d3c0c5e Merge branch 'worktree-agent-ae13fcbf826d32548' 2026-09-05 22:42:11 +01:00
Hugh Kaznowski 48c08328af Merge branch 'worktree-agent-aa829d0368bca7e78' 2026-09-05 22:41:54 +01:00
Hugh KaznowskiandClaude Sonnet 5 e397f9fbdb Unify param declarations: metrics reference params by name (ticket 037) 
Collapse the Params-panel ParamDeclaration and the metric-embedded
RegexToMetricPrimitiveMatcher into a single ParamDeclaration type
(name, regex/capture-group-or-json-field-path, kind). MetricDeclaration
now references params by name via label_params: Vec<String> and
value_param: Option<String> instead of embedding its own extractor per
param, so a param declared once in the Params panel can be reused as a
label or value by any number of metrics. Declaring a metric that
references an undeclared param name is now rejected at declare-time.

Backend:
- src/data_value.rs: unified ParamDeclaration struct; MetricDeclaration
  data value now carries label_params/value_param instead of a Vec of
  inline matchers.
- src/engine.rs: DeclareParamRequest is now an alias for
  ParamDeclaration; DeclareMetricRequest carries label_params/
  value_param; validate_metric_request checks referenced param names
  against the declared-param store; apply_metric_to_log resolves
  label_params/value_param against a per-ingest resolved-param map
  instead of reading an inline matcher list.
- src/profile.rs: MetricDeclarationRecord carries label_params/
  value_param plus a private, serialize-skipped legacy_params field
  that only ever gets populated by deserializing a pre-037 profile
  file's old "params" key. ProfileStore::load migrates any such legacy
  metric into the unified shape (reusing an existing same-shaped bare
  param by name, or creating one implicitly named
  "<metric_name>.<param_name>"), then persists the migrated store so
  the on-disk file only carries the legacy shape once. Migration is
  idempotent (deterministic naming) and covered by dedicated tests.
- src/api.rs / src/dto.rs: MetricDeclarationDto and the declare/update
  metric response bodies updated to the new shape.

Frontend:
- webapp/src/lib/api.ts: MetricDeclaration type now has label_params/
  value_param; MetricParamMatcher/ParamRole types removed.
- webapp/src/components/Metrics.tsx: the declaration form now fetches
  declared params and lets you check off label params / pick a value
  param from that list, instead of re-entering a regex/capture-group/
  kind per metric param.

Leaves ticket 038 (duration as a derived param kind) and 039 (param
query resolving declared params) untouched, as scoped.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:41:14 +01:00
Hugh KaznowskiandClaude Sonnet 5 ef60dd267b Unify PromQL rate() and alert Threshold/Ratio windowing (ticket 040) 
Extract the trailing-window rate/count implementation into a new src/rate.rs,
shared by PromQL's rate() (engine::evaluate_promql, via a thin i64<->Timestamp
adapter) and alert Threshold/Ratio conditions (alerts::evaluate_condition, via
the new incremental RateWindow), replacing alerts' bespoke VecDeque+prune loop.
Both now agree on what "windowed rate" means for the same data, verified by a
new regression test (engine::tests::dashboard_rate_and_alert_threshold_agree_on_same_windowed_data).

Behavior change: alert Threshold/Ratio windows are now backfilled once, on
first evaluation, from already-ingested logs within their own trailing window
(bounded cost, not a full history rescan) - a rule created mid-session, or a
process that replays existing log content (e.g. --file tailing, which rereads
the whole file from the start on every process start), no longer starts with
a silently empty window. A bare restart with no replayed history (e.g. stdin
ingestion) still starts empty, since there is genuinely no log history to
backfill from in that case - alert state itself remains in-memory only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:34:13 +01:00
Hugh KaznowskiandClaude Sonnet 5 9f84be26ef Fix sum()/avg() merging sparse across staggered raw timestamps (ticket 042) 
merge_series_exact required an exact-millisecond timestamp match to combine
sub-series, so sum(requests) by (path)-style queries over raw occurrence
counts produced one point per underlying (label, timestamp) pair instead of
a meaningful combined total, since real occurrence timestamps across
different label groups essentially never coincide. Added a regression test
confirming the sparse behavior, then replaced the merge with
merge_series_windowed: a single-sub-series group is passed through
unchanged (preserving existing single-label-group resolution exactly), and
groups with multiple sub-series are bucketed into fixed 1s windows before
summing/averaging, mirroring the fixed-width bucketing rate() already uses.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:27:59 +01:00
Hugh KaznowskiandClaude Sonnet 5 af7a4f1327 Reject sum()/avg() over string-typed metrics; add explicit count() 
`coerce_to_f64` (src/engine.rs) mapped every MetricPrimitive::String to a
flat 1.0, so sum()/avg() over a string-kind metric silently became an
occurrence count with no signal that the aggregate didn't mean what it
looked like it meant.

evaluate_promql now looks up the target metric's declared value kind (the
kind of its Value-role param, defaulting to Number when it has none) and
rejects sum()/avg() with a clear, actionable error when that kind is
String. sum(rate(...))/avg(rate(...)) are exempt since rate() only counts
occurrences per window and never reads the coerced value. Also add a real
count() PromQL aggregation (parser + evaluator) so occurrence-counting is
something a user opts into by name, matching the ticket's preferred
"count() or numeric metric" error message and the existing
sum()/avg()/rate()/histogram()/gauge() aggregation design.

Audited existing fixtures/tests for sum()/avg() over string-kind metrics
that this would newly reject: none found — the repo's only sum()/avg()
usage in tests and docs is over the "requests" occurrence-counter metric,
which has no Value-role param and so is Number-kind, unaffected by this
change.

EngineResponse::Promql already carried Result<Vec<PromqlSeriesDto>,
String> for parser errors, so evaluate_promql now returns that same
Result and the new error flows through the existing 400 Bad Request path
in run_promql_query with no API changes needed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:27:57 +01:00
Hugh KaznowskiandClaude Sonnet 5 db6feff76c Support by() label grouping for histogram() queries 
sum/avg/rate already grouped series by the query's by_labels, but
histogram() collapsed every label combination into one histogram
regardless of by(), silently discarding the grouping. evaluate_promql's
Histogram branch now groups series by by_labels first (reusing the same
project_labels mechanism as sum/avg), then buckets each group's values
independently, so histogram(metric) by (label) yields one histogram per
label group carrying that group's labels alongside `le`. An empty
by_labels still collapses to a single histogram, matching prior
behavior exactly, so sum/avg/rate and unqualified histogram() queries
are unaffected.

The dashboard panel UI needed no change: PromQL is entered as free
text and PanelChart's seriesLabel()/legend rendering already renders
arbitrary label maps generically, so grouped histogram series display
correctly out of the box.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
 2026-09-05 22:24:43 +01:00
Hugh Kaznowski 930a2f2536 Add tickets to improve metric abstractions 2026-09-05 22:20:03 +01:00
67 changed files with 12311 additions and 4994 deletions
Expand all files Collapse all files
.gitignore
+1
View File
@@ -1,3 +1,4 @@
/target /target
target/ target/
.claude/worktrees/ .claude/worktrees/
profile.json.gz
CLAUDE.md
+45
View File
@@ -0,0 +1,45 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
ottyl turns a piped command's stdout (or a tailed file) into a local observability dashboard: searchable log timeline, ad-hoc metrics extracted from log lines via regex, and Grafana-style dashboards — all served from a single self-contained binary for the lifetime of that process. It can also poll a Prometheus/OpenMetrics endpoint alongside log ingestion. See `README.md` for the user-facing feature list and CLI usage.
## Commands
```bash
make build # builds webapp (npm install + vite build) then the release Rust binary; webapp/dist is embedded into the binary
make dev # frontend dev server with hot reload, proxying API calls to a separately running `cargo run` backend (see webapp/vite.config.ts)
make test # cargo test + frontend build (frontend has no test suite, just `tsc -b && vite build`)
make lint # cargo clippy --all-targets + oxlint (webapp)
make run # build then run ./target/release/ottyl
```
- Single backend test: `cargo test <test_name>`
- Backend tests live inline in each `src/*.rs` file (`#[cfg(test)] mod tests`), not in a separate `tests/` dir.
- The webapp has no unit tests — `npm run build` (tsc + vite) is the only frontend check besides lint.
- `webapp/dist` must exist before `cargo build` will produce a working binary (it's embedded via `fs_embed!`), so always build the frontend first — `make build` handles ordering; don't run `cargo build` alone after frontend changes without rebuilding `webapp/dist`.
## Architecture
Backend: Rust (axum + tokio), single binary, in `src/`. Frontend: React 19 + TypeScript + Tailwind + Recharts, in `webapp/`, built with Vite and embedded into the binary at build time (`src/api.rs`'s `WEBAPP_DIST` via `fs_embed!("webapp/dist")`).
**Data flow:** `main.rs` wires everything together at startup — loads `Settings`, spawns the engine actor task, replays the active profile's param/metric declarations into it (before ingest starts, since there's no way to reprocess earlier lines later), then starts ingest (stdin/file tailing and/or Prometheus polling) and the axum HTTP server.
- **`engine.rs`** (largest file, ~3400 lines) — the actor owning all live state: ingested log lines, declared params/metrics, computed metric buckets, alert evaluation. Runs as a single tokio task; all access is through `EngineHandle` sending `EngineMessage` over an mpsc channel and awaiting a response via oneshot — there is no shared mutable engine state accessed directly from request handlers. This is the file to read first to understand how declarations, filtering, and bucketing actually work.
- **`api.rs`** — axum routes and HTTP-facing `ApiState` (holds `EngineHandle` + `SettingsHandle`). Translates HTTP requests into `EngineMessage`s and DTOs back into JSON. Also serves the embedded webapp.
- **`settings.rs` / `profile.rs`** — persisted configuration. A **profile** is per-app: parsing rules (param/metric declarations) and dashboards, saved so they're there again next time that app runs through ottyl (see README's "Profiles"). Declarations are persisted here so the engine can replay them on restart; `api.rs` handlers that mutate engine state (e.g. `declare_metric`) call a corresponding persist method here so the two stay in sync — this dual-write is a recurring pattern (see `ApiState::persist_param_declaration` / `persist_metric_declaration` in `api.rs`), and adding a new declaration-mutating endpoint means updating both sides plus the load-time replay path.
- **`filter_parser.rs`** (grammar in `filter_parser.pest`, via `pest`) + **`filter_datetime.rs`** — parse the search/filter bar syntax (regex matches, metric comparisons like `latency_ms[value]<200`, relative/absolute datetimes).
- **`promql.rs`** — parses and evaluates the PromQL-style queries backing dashboard panels.
- **`logfmt.rs` / `ingest.rs`** — line ingestion: tailing stdin/a file, structured logfmt-style extraction, and polling a Prometheus/OpenMetrics `--metrics-url`.
- **`alerts.rs`** — alert rule evaluation and ntfy delivery.
- **`dto.rs`** — shared request/response types between `api.rs` and the frontend's `webapp/src/lib/api.ts`; **`data_value.rs`** / **`metric_primitive.rs`** — the value/type model for params and metrics; **`index_key.rs`** / **`rate.rs`** — bucket indexing and rate computation helpers.
Frontend structure: `webapp/src/App.tsx` is the shell; `webapp/src/components/` has one component per feature area (`FilterBar`, `LogTimeline`, `Metrics`, `Params`, `Dashboards`, `Alerts`, `Docs`, `NavMenu`, ...) that map closely to the backend concepts above. `webapp/src/lib/api.ts` is the typed client for the Rust API; `webapp/src/lib/polling.ts` handles refresh polling; `webapp/src/lib/themes.ts` covers the style-preset/CSS system.
## Conventions
- **`tickets/`** holds scoped, written-up work items (problem statement, pointers to exact call sites with `file.rs:line` references, and an explicit scope section) — check here for planned/in-flight work before starting something that might overlap. Tickets are removed from this directory once implemented (see recent commit history — e.g. "Remove ticket 047 (implemented: ...)").
- Declaration-related features (params, metrics) consistently need three coordinated changes: an `EngineMessage` variant + handler in `engine.rs`, a persistence method in `settings.rs`/`profile.rs`, and an `api.rs` route wiring them together (plus a webapp UI affordance). Grep for an existing similar declaration (e.g. how metric declaration is wired) as the template before adding a new one.
- Doc comments in this codebase tend to explain *why* a non-obvious choice was made (e.g. ordering constraints, a workaround, a past ticket's rationale) rather than restating what the code does — match that style rather than adding descriptive comments.
Cargo.lock
Generated
+438
View File
@@ -70,6 +70,12 @@ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.61.2",
] ]
[[package]]
name = "anyhow"
version = "1.0.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]] [[package]]
name = "atomic-waker" name = "atomic-waker"
version = "1.1.2" version = "1.1.2"
@@ -200,6 +206,7 @@ dependencies = [
"iana-time-zone", "iana-time-zone",
"js-sys", "js-sys",
"num-traits", "num-traits",
"serde",
"wasm-bindgen", "wasm-bindgen",
"windows-link", "windows-link",
] ]
@@ -265,6 +272,46 @@ dependencies = [
"libc", "libc",
] ]
[[package]]
name = "crossbeam-utils"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a31eee39dddec8330830986fcd7625edb5a24ec90ea038215273bbc3adb08ac6"
[[package]]
name = "darling"
version = "0.24.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed17f5901b6630b993ca003def43f2f8ef4014fc13b047b57aad617ff32bc2ec"
dependencies = [
"darling_core",
"darling_macro",
]
[[package]]
name = "darling_core"
version = "0.24.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6837e2cf7485aaae18f86181d2f0e9a7ed297a025e220aeabf63fdebd3a2ddff"
dependencies = [
"ident_case",
"proc-macro2",
"quote",
"strsim",
"syn 3.0.5",
]
[[package]]
name = "darling_macro"
version = "0.24.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785"
dependencies = [
"darling_core",
"quote",
"syn 3.0.5",
]
[[package]] [[package]]
name = "directories" name = "directories"
version = "6.0.0" version = "6.0.0"
@@ -297,6 +344,18 @@ dependencies = [
"syn 3.0.5", "syn 3.0.5",
] ]
[[package]]
name = "dyn-clone"
version = "1.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "equivalent"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]] [[package]]
name = "errno" name = "errno"
version = "0.3.14" version = "0.3.14"
@@ -355,6 +414,21 @@ dependencies = [
"walkdir", "walkdir",
] ]
[[package]]
name = "futures"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a31d2a3fbaaeb2af2368bbdd904aa8e812d3c04a1ee10d3171f52d556e5d0a3"
dependencies = [
"futures-channel",
"futures-core",
"futures-executor",
"futures-io",
"futures-sink",
"futures-task",
"futures-util",
]
[[package]] [[package]]
name = "futures-channel" name = "futures-channel"
version = "0.3.34" version = "0.3.34"
@@ -362,6 +436,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4" checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4"
dependencies = [ dependencies = [
"futures-core", "futures-core",
"futures-sink",
] ]
[[package]] [[package]]
@@ -370,6 +445,40 @@ version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
[[package]]
name = "futures-executor"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "031b47cf1a3c6cc8bc2fc76cd437f521619387907d469316e7c0bc278f1f5432"
dependencies = [
"futures-core",
"futures-task",
"futures-util",
]
[[package]]
name = "futures-io"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed"
[[package]]
name = "futures-macro"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
]
[[package]]
name = "futures-sink"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d"
[[package]] [[package]]
name = "futures-task" name = "futures-task"
version = "0.3.34" version = "0.3.34"
@@ -382,12 +491,32 @@ version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc"
dependencies = [ dependencies = [
"futures-channel",
"futures-core", "futures-core",
"futures-io",
"futures-macro",
"futures-sink",
"futures-task", "futures-task",
"memchr",
"pin-project-lite", "pin-project-lite",
"slab", "slab",
] ]
[[package]]
name = "generator"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b3b854b0e584ead1a33f18b2fcad7cf7be18b3875c78816b753639aa501513ae"
dependencies = [
"cc",
"cfg-if",
"libc",
"log",
"rustversion",
"windows-link",
"windows-result",
]
[[package]] [[package]]
name = "getrandom" name = "getrandom"
version = "0.2.17" version = "0.2.17"
@@ -415,6 +544,12 @@ dependencies = [
"wasm-bindgen", "wasm-bindgen",
] ]
[[package]]
name = "hashbrown"
version = "0.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
[[package]] [[package]]
name = "heck" name = "heck"
version = "0.5.0" version = "0.5.0"
@@ -639,6 +774,12 @@ dependencies = [
"zerovec", "zerovec",
] ]
[[package]]
name = "ident_case"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
[[package]] [[package]]
name = "idna" name = "idna"
version = "1.1.0" version = "1.1.0"
@@ -679,6 +820,18 @@ dependencies = [
"quote", "quote",
] ]
[[package]]
name = "indexmap"
version = "2.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855"
dependencies = [
"equivalent",
"hashbrown",
"serde",
"serde_core",
]
[[package]] [[package]]
name = "ipnet" name = "ipnet"
version = "2.12.1" version = "2.12.1"
@@ -708,6 +861,23 @@ dependencies = [
"wasm-bindgen", "wasm-bindgen",
] ]
[[package]]
name = "lazy_static"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
[[package]]
name = "left-right"
version = "0.11.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8bc015ded5d9b3054dbbdb63332cdd6ee42352ccef19e911e25117490e2f48ee"
dependencies = [
"crossbeam-utils",
"loom",
"slab",
]
[[package]] [[package]]
name = "libc" name = "libc"
version = "0.2.189" version = "0.2.189"
@@ -750,12 +920,34 @@ version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
[[package]]
name = "loom"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca"
dependencies = [
"cfg-if",
"generator",
"scoped-tls",
"tracing",
"tracing-subscriber",
]
[[package]] [[package]]
name = "lru-slab" name = "lru-slab"
version = "0.1.2" version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "matchers"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
dependencies = [
"regex-automata",
]
[[package]] [[package]]
name = "matchit" name = "matchit"
version = "0.8.4" version = "0.8.4"
@@ -774,6 +966,12 @@ version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "minimal-lexical"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
[[package]] [[package]]
name = "mio" name = "mio"
version = "1.2.3" version = "1.2.3"
@@ -785,6 +983,25 @@ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.61.2",
] ]
[[package]]
name = "nom"
version = "7.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
dependencies = [
"memchr",
"minimal-lexical",
]
[[package]]
name = "nu-ansi-term"
version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.61.2",
]
[[package]] [[package]]
name = "num-traits" name = "num-traits"
version = "0.2.19" version = "0.2.19"
@@ -816,6 +1033,7 @@ checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
name = "ottyl" name = "ottyl"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"anyhow",
"axum", "axum",
"chrono", "chrono",
"clap", "clap",
@@ -824,10 +1042,14 @@ dependencies = [
"http-body-util", "http-body-util",
"humantime", "humantime",
"include_dir", "include_dir",
"left-right",
"nom",
"pest", "pest",
"pest_derive", "pest_derive",
"petname",
"regex", "regex",
"reqwest", "reqwest",
"rmcp",
"serde", "serde",
"serde_json", "serde_json",
"tempfile", "tempfile",
@@ -858,6 +1080,12 @@ dependencies = [
"windows-link", "windows-link",
] ]
[[package]]
name = "pastey"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
[[package]] [[package]]
name = "percent-encoding" name = "percent-encoding"
version = "2.3.2" version = "2.3.2"
@@ -906,6 +1134,27 @@ dependencies = [
"pest", "pest",
] ]
[[package]]
name = "petname"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bfb4653a82ec48e6d324c788597aadf46449bfc7b8e76d2a03b513a944325c88"
dependencies = [
"petname-macros",
"rand",
]
[[package]]
name = "petname-macros"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c406fb544296f598ccf69b82bcf1b12440c8f23ebf75946105f86304f16eb2"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
]
[[package]] [[package]]
name = "phf" name = "phf"
version = "0.11.3" version = "0.11.3"
@@ -1065,6 +1314,26 @@ dependencies = [
"thiserror", "thiserror",
] ]
[[package]]
name = "ref-cast"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e440fb4e4b4147295338efb76001ab9e4efc0e5839df2c47fc5ac2381d365c3"
dependencies = [
"ref-cast-impl",
]
[[package]]
name = "ref-cast-impl"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
]
[[package]] [[package]]
name = "regex" name = "regex"
version = "1.13.1" version = "1.13.1"
@@ -1146,6 +1415,41 @@ dependencies = [
"windows-sys 0.52.0", "windows-sys 0.52.0",
] ]
[[package]]
name = "rmcp"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42b6914fac0be956fe704a38239c3f44a9f841d1b06a5713d2f638065593f5b5"
dependencies = [
"chrono",
"futures",
"indexmap",
"pastey",
"pin-project-lite",
"rmcp-macros",
"schemars",
"serde",
"serde_json",
"thiserror",
"tokio",
"tokio-util",
"tracing",
"uuid",
]
[[package]]
name = "rmcp-macros"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cdf1c49bd4d52014b94db0877410db273c2008f01628b0252a2e9460ad9b7fda"
dependencies = [
"darling",
"proc-macro2",
"quote",
"serde_json",
"syn 3.0.5",
]
[[package]] [[package]]
name = "rustc-hash" name = "rustc-hash"
version = "2.1.3" version = "2.1.3"
@@ -1221,6 +1525,38 @@ dependencies = [
"winapi-util", "winapi-util",
] ]
[[package]]
name = "schemars"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a"
dependencies = [
"chrono",
"dyn-clone",
"ref-cast",
"schemars_derive",
"serde",
"serde_json",
]
[[package]]
name = "schemars_derive"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d98c67716b46af2f0b8cf752abc930f6f9aecfbf671ecfb531db8a31dbe4e2ba"
dependencies = [
"proc-macro2",
"quote",
"serde_derive_internals",
"syn 3.0.5",
]
[[package]]
name = "scoped-tls"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294"
[[package]] [[package]]
name = "scopeguard" name = "scopeguard"
version = "1.2.0" version = "1.2.0"
@@ -1257,6 +1593,17 @@ dependencies = [
"syn 3.0.5", "syn 3.0.5",
] ]
[[package]]
name = "serde_derive_internals"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f852137cce035d6a4df67ccce505ff6b3e9fd3a10e3e52b24dc71e650bb1a9bd"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
]
[[package]] [[package]]
name = "serde_json" name = "serde_json"
version = "1.0.151" version = "1.0.151"
@@ -1293,6 +1640,15 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "sharded-slab"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6"
dependencies = [
"lazy_static",
]
[[package]] [[package]]
name = "shlex" name = "shlex"
version = "2.0.1" version = "2.0.1"
@@ -1430,6 +1786,15 @@ dependencies = [
"syn 3.0.5", "syn 3.0.5",
] ]
[[package]]
name = "thread_local"
version = "1.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070"
dependencies = [
"cfg-if",
]
[[package]] [[package]]
name = "tinystr" name = "tinystr"
version = "0.8.4" version = "0.8.4"
@@ -1493,6 +1858,20 @@ dependencies = [
"tokio", "tokio",
] ]
[[package]]
name = "tokio-util"
version = "0.7.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52"
dependencies = [
"bytes",
"futures-core",
"futures-sink",
"libc",
"pin-project-lite",
"tokio",
]
[[package]] [[package]]
name = "tower" name = "tower"
version = "0.5.3" version = "0.5.3"
@@ -1547,9 +1926,21 @@ checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
dependencies = [ dependencies = [
"log", "log",
"pin-project-lite", "pin-project-lite",
"tracing-attributes",
"tracing-core", "tracing-core",
] ]
[[package]]
name = "tracing-attributes"
version = "0.1.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]] [[package]]
name = "tracing-core" name = "tracing-core"
version = "0.1.36" version = "0.1.36"
@@ -1557,6 +1948,36 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
dependencies = [ dependencies = [
"once_cell", "once_cell",
"valuable",
]
[[package]]
name = "tracing-log"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3"
dependencies = [
"log",
"once_cell",
"tracing-core",
]
[[package]]
name = "tracing-subscriber"
version = "0.3.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
dependencies = [
"matchers",
"nu-ansi-term",
"once_cell",
"regex-automata",
"sharded-slab",
"smallvec",
"thread_local",
"tracing",
"tracing-core",
"tracing-log",
] ]
[[package]] [[package]]
@@ -1607,6 +2028,23 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.26.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5772d71c9be8a8a6ac2117d949c5b224c1b72241bb611d9a3012edcf8af7812"
dependencies = [
"getrandom 0.4.3",
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "valuable"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
[[package]] [[package]]
name = "walkdir" name = "walkdir"
version = "2.5.0" version = "2.5.0"
Cargo.toml
+6 -1
View File
@@ -10,6 +10,7 @@ serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
clap = { version = "4", features = ["derive"] } clap = { version = "4", features = ["derive"] }
regex = "1" regex = "1"
nom = "7"
pest = "2" pest = "2"
pest_derive = "2" pest_derive = "2"
chrono = "0.4" chrono = "0.4"
@@ -17,7 +18,11 @@ humantime = "2"
directories = "6.0.0" directories = "6.0.0"
fs-embed = "0.2.4" fs-embed = "0.2.4"
include_dir = "0.7.4" include_dir = "0.7.4"
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] } reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
petname = { version = "3.2.0", default-features = false, features = ["default-rng", "default-words"] }
rmcp = { version = "3.2.0", default-features = false, features = ["server", "macros", "transport-io"] }
anyhow = "1"
left-right = "0.11.8"
[dev-dependencies] [dev-dependencies]
tempfile = "3.27.0" tempfile = "3.27.0"
IDEAS.md
+12
View File
@@ -0,0 +1,12 @@
Under a normal log ingestion params are usually auto-detected.
Sometimes, however it's necessary to extract a param - that can be done with a regex: If a message has a non empty or missing capture for the target group, then it has a param value.
On top of that, it may be easier to capture params dependent on these other params - say you have a string of a capture group you are interested in, and you want to extract values out of it. That should be possible as well as a regex patten on top of messages with a string param.
Sometimes a string param (or even message) may be json. In which case we may want to extract params with `jq` syntax.
There is lots of test-helper code that is simply a function stub with ignored parameters and a different scope. We don't need any of those ever. You can change the scope of some functions for the tests. Preferably you change the tests instead.
The timeline still does not reflect the dropdown. Either use a toast with a reason why it can't while setting the dropdown to what it currently was, or have some kind of loading indicator if it's a loading issue.
JSON formatting on frontend should auto-collapsed not open
SKILL.md
+154
View File
@@ -0,0 +1,154 @@
---
name: ottyl
description: Query and configure a running ottyl observability instance (logs, params, metrics, dashboards, settings, alerts, profiles) via its MCP server.
---
# ottyl
ottyl turns a piped command's stdout into a searchable log timeline plus ad-hoc metrics and
dashboards, served from a single process. This doc covers how an agent connects to and drives a
*running* instance.
Everything the web UI can do to change the running instance's state is exposed as an MCP tool —
declarations, dashboards, settings, alerts, profiles. Chart panels aren't screenshotted: use
`render_panel` / `render_dashboard` to get a chart's underlying series data back as JSON, so you
can reason about it directly instead of needing a browser.
## Find a session
Each `ottyl` process registers itself under a name (random, e.g. `comic-gazelle`, unless started
with `--name <name>`). List known sessions:
```
ottyl sessions
# comic-gazelle port=51423 pid=8421
```
## Connect an MCP client
Start the MCP server (a separate short-lived process) pointed at that session, and register it
with your MCP client (e.g. Claude Code's `.mcp.json`):
```
ottyl mcp --session comic-gazelle
# or, if you already know the port:
ottyl mcp --port 51423
```
This is a thin proxy onto the target instance's HTTP API — each tool below maps to one or more
`/api/...` routes.
## Get the latest SKILL.md
This file is compiled into the `ottyl` binary, so a copy you have on disk can lag behind the
version a running instance actually serves (e.g. after an upgrade). A running instance always
serves its own current copy read-only over HTTP — fetch it directly rather than trusting a stale
local file:
```
curl http://localhost:<port>/SKILL.md
```
Use the port from `ottyl sessions` (or the `--port`/`--session` you passed to `ottyl mcp`). There
is no CLI subcommand for this and no way to update it at runtime — the served content only changes
when the binary is rebuilt with a new `SKILL.md`.
## Tools
Most tools that operate on a profile take an optional `profile` argument; omit it to use the
instance's active profile.
### Logs, timeline, stats
- **`query_logs`** — count or read log lines matching a FilterBar query, with time/sequence range
and sort options.
```json
{ "filter": "regex(\"error\")", "count_only": true }
```
- **`get_timeline`** — bucketed log-count histogram matching an optional filter (the data behind
the Log Timeline chart).
- **`get_stats`** — overall ingest stats: log count, sequence/timestamp bounds, ingest lag.
### Params
- **`list_params`** — list declared params (including auto-derived ones).
- **`declare_param`** — extract a named value from matching log lines.
```json
{ "name": "status_code", "message_match_regex": "status=(\\d+)", "capture_group": "1", "kind": "number" }
```
- **`clear_params`** — clear all declared params and re-derive auto params from retained logs.
### Metrics
- **`list_metrics`** — list declared metrics.
- **`declare_metric`** — bucket event counts/values by label, derived from matching log lines.
```json
{ "name": "errors_total", "message_match_regex": "level=error", "label_params": ["status_code"] }
```
- **`update_metric`** — update an existing metric declaration in place (`name` plus the same
fields as `declare_metric`).
- **`delete_metric`** — delete a declared metric by name.
- **`get_metric_bucket_values`** — read a metric's raw bucket values, optionally filtered by
label (`labels: "status_code=500"`) and time/sequence range.
### PromQL
- **`run_promql_query`** — evaluate a PromQL-style query directly, returning raw series data
(`labels` + timestamped `points`) — the same evaluation dashboards use.
```json
{ "promql": "rate(errors_total[5m])" }
```
### Dashboards & panels
- **`list_dashboards`** — list dashboards saved in a profile.
- **`create_dashboard`** — create an empty dashboard.
```json
{ "name": "Errors" }
```
- **`get_dashboard`** — read a dashboard's panel definitions (promql/chart_type per panel, not
rendered data).
- **`delete_dashboard`** — delete a dashboard.
- **`add_panel`** — add a panel, or update one in place by `panel_id`.
```json
{ "dashboard_id": "<id>", "title": "Errors over time", "promql": "rate(errors_total[5m])", "chart_type": "line" }
```
- **`delete_panel`** — remove a panel from a dashboard by `panel_id`.
- **`render_panel`** — resolve one panel's promql, evaluate it, and return `{ panel, series }` as
JSON — this is how to "see" a chart without a browser.
```json
{ "dashboard_id": "<id>", "panel_id": "<id>" }
```
- **`render_dashboard`** — same as `render_panel` but for every panel in a dashboard at once,
returning `{ dashboard, panels: [{ panel, series }, ...] }`.
### Settings & theming
- **`get_settings`** / **`update_settings`** — read/update color scheme, active profile,
`max_logs`, custom theme, ntfy topic/server.
- **`list_css`** — list imported/linked custom CSS theme entries.
- **`import_css`** — import a CSS theme by file content.
- **`link_css`** — link an external CSS file/URL as a theme instead of importing content.
- **`remove_css`** / **`get_css_content`** — remove a CSS entry / read its raw content by id.
### Alerts
- **`list_alert_rules`** — list configured alert rules.
- **`create_alert_rule`** / **`update_alert_rule`** — `condition` is one of:
- `{"kind":"immediate","query":"..."}`
- `{"kind":"threshold","query":"...","op":"gt|lt|ge|le|eq|ne","threshold":N,"window_ms":N}`
- `{"kind":"newvalue","field_regex":"...","capture_group":"..."}`
- `{"kind":"ratio","query_a":"...","query_b":"...","op":"...","ratio":N,"window_ms":N}`
- **`delete_alert_rule`** — delete a rule by id.
- **`test_alert_notification`** — send a test ntfy notification to verify delivery config.
### Profiles
- **`list_profiles`** — list saved profiles with summary counts.
- **`create_profile`** — create a new, empty profile.
```json
{ "name": "my-service" }
```
- **`delete_profile`** — delete a profile by name.
- **`activate_profile`** — activate a profile, swapping in its declarations and dashboards as the
running instance's live state.
src/dto.rs → src/api/dto.rs
+16 -45
View File
@@ -1,8 +1,8 @@
use crate::alerts::AlertCondition; use crate::engine::alerts::AlertCondition;
use crate::data_value::{MetricPrimitiveKind, RegexToMetricPrimitiveMatcher}; use crate::engine::data_value::MetricPrimitiveKind;
use crate::metric_primitive::MetricPrimitive; use crate::engine::metric_primitive::MetricPrimitive;
use crate::profile::{ChartType, Dashboard, Panel, Profile}; use crate::config::profile::{ChartType, Dashboard, Panel, Profile};
use crate::settings::{ColorScheme, CssEntry, Settings}; use crate::config::{ColorScheme, CssEntry, Settings};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::BTreeMap; use std::collections::BTreeMap;
@@ -11,7 +11,6 @@ pub struct LogEntryDto {
pub sequence: u64, pub sequence: u64,
pub timestamp_ms: i64, pub timestamp_ms: i64,
pub text: String, pub text: String,
/// Only params whose regex matched (and parsed as the declared kind) are present — omitted rather than sent as null, so the frontend can render a blank for anything missing.
pub params: BTreeMap<String, MetricPrimitive>, pub params: BTreeMap<String, MetricPrimitive>,
} }
@@ -20,19 +19,21 @@ pub struct ParamDeclarationDto {
pub name: String, pub name: String,
pub message_match_regex: String, pub message_match_regex: String,
pub capture_group: String, pub capture_group: String,
/// See `crate::data_value::RegexToMetricPrimitiveMatcher::json_field_path`.
pub json_field_path: Option<String>, pub json_field_path: Option<String>,
pub kind: MetricPrimitiveKind, pub kind: MetricPrimitiveKind,
pub elapsed_since_last: bool,
pub auto_derived: bool,
#[serde(default)]
pub reserved: bool,
} }
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
pub struct MetricDeclarationDto { pub struct MetricDeclarationDto {
pub name: String, pub name: String,
pub message_match_regex: String, pub message_match_regex: String,
pub params: Vec<RegexToMetricPrimitiveMatcher>, pub label_params: Vec<String>,
/// See `crate::engine::DeclareMetricRequest::matched_labels`. pub value_param: Option<String>,
pub matched_labels: Option<BTreeMap<String, String>>, pub matched_labels: Option<BTreeMap<String, String>>,
/// See `crate::engine::DeclareMetricRequest::unmatched_labels`.
pub unmatched_labels: Option<BTreeMap<String, String>>, pub unmatched_labels: Option<BTreeMap<String, String>>,
} }
@@ -49,19 +50,8 @@ pub struct StatsDto {
pub last_sequence: Option<u64>, pub last_sequence: Option<u64>,
pub first_timestamp_ms: Option<i64>, pub first_timestamp_ms: Option<i64>,
pub last_timestamp_ms: Option<i64>, pub last_timestamp_ms: Option<i64>,
/// Ticket 011: a running, *approximate* estimate of the in-memory heap
/// held by currently-ingested logs, in bytes. "Approximate" because it
/// only counts each `IngestedLine::text`'s UTF-8 byte length — it does
/// not account for `String`'s allocator overhead/capacity slack, the
/// `BTreeMap`/`IndexKey` node overhead the text is stored under, or any
/// derived data (bucket values, params) the line produces elsewhere in
/// the index. That's judged the best cheap proxy for "how much memory is
/// this log's text costing us": it's O(1) to compute per log (`text.len()`,
/// already an interned byte count for UTF-8 `String`), moves in lockstep
/// with the dominant cost (raw log text, which is usually much larger
/// than the fixed per-entry bookkeeping), and needs no per-entry
/// bookkeeping beyond what eviction already tracks.
pub approx_bytes_total: u64, pub approx_bytes_total: u64,
pub ingest_lag_high_watermark_ms: i64,
} }
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
@@ -70,8 +60,9 @@ pub struct LogsParams {
pub start_ms: Option<i64>, pub start_ms: Option<i64>,
pub end_ms: Option<i64>, pub end_ms: Option<i64>,
pub limit: Option<usize>, pub limit: Option<usize>,
/// Raw FilterBar query string, e.g. `regex("error") and sequence<5`.
pub filter: Option<String>, pub filter: Option<String>,
pub sort_by: Option<String>,
pub sort_missing_fallback: Option<bool>,
} }
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
@@ -79,11 +70,9 @@ pub struct TimelineParams {
pub start_ms: Option<i64>, pub start_ms: Option<i64>,
pub end_ms: Option<i64>, pub end_ms: Option<i64>,
pub bucket_ms: Option<i64>, pub bucket_ms: Option<i64>,
/// Same FilterBar query string as `LogsParams::filter`; only matching logs are counted into timeline buckets.
pub filter: Option<String>, pub filter: Option<String>,
} }
/// One (sequence, timestamp, value) occurrence within a metric's bucket for a given label combination.
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
pub struct BucketValueDto { pub struct BucketValueDto {
pub labels: BTreeMap<String, MetricPrimitive>, pub labels: BTreeMap<String, MetricPrimitive>,
@@ -92,7 +81,6 @@ pub struct BucketValueDto {
pub value: MetricPrimitive, pub value: MetricPrimitive,
} }
/// `labels` is a comma-separated `key=value` list (not repeated params) since axum's `Query` extractor needs a fixed set of field names.
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct BucketValuesParams { pub struct BucketValuesParams {
pub labels: Option<String>, pub labels: Option<String>,
@@ -102,7 +90,6 @@ pub struct BucketValuesParams {
pub limit: Option<usize>, pub limit: Option<usize>,
} }
/// `labels` uses plain `String` (already Display-formatted) since charts just need something to show; `value` is always `f64` for plottability.
#[derive(Debug, Clone, Serialize, PartialEq)] #[derive(Debug, Clone, Serialize, PartialEq)]
pub struct PromqlSeriesDto { pub struct PromqlSeriesDto {
pub labels: BTreeMap<String, String>, pub labels: BTreeMap<String, String>,
@@ -116,27 +103,22 @@ pub struct PromqlParams {
pub end_ms: Option<i64>, pub end_ms: Option<i64>,
} }
/// JSON error body returned by API handlers on a 4xx/5xx response.
#[derive(Debug, Serialize)] #[derive(Debug, Serialize)]
pub struct ErrorDto { pub struct ErrorDto {
pub error: String, pub error: String,
} }
/// Every imported CSS entry is a selectable theme (ticket 020) — there's no separate
/// always-on-overlay mode to opt into.
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct ImportCssRequest { pub struct ImportCssRequest {
pub filename: String, pub filename: String,
pub content: String, pub content: String,
} }
/// A reference to an external CSS path/URL that ottyl doesn't copy or own.
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct LinkCssRequest { pub struct LinkCssRequest {
pub path_or_url: String, pub path_or_url: String,
} }
/// `id` is the real id for `Imported` entries or a stable id derived from the path/URL for `Linked` ones, so the frontend can key both kinds uniformly (only `Imported` entries are deletable/content-fetchable).
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
pub struct CssEntryDto { pub struct CssEntryDto {
pub id: String, pub id: String,
@@ -162,7 +144,7 @@ impl From<&CssEntry> for CssEntryDto {
path_or_url: None, path_or_url: None,
}, },
CssEntry::Linked { path_or_url } => CssEntryDto { CssEntry::Linked { path_or_url } => CssEntryDto {
id: crate::settings::css_entry_id(entry), id: crate::config::css_entry_id(entry),
kind: CssEntryKind::Linked, kind: CssEntryKind::Linked,
filename: None, filename: None,
path_or_url: Some(path_or_url.clone()), path_or_url: Some(path_or_url.clone()),
@@ -171,18 +153,13 @@ impl From<&CssEntry> for CssEntryDto {
} }
} }
/// `css_entries` is omitted here since it has its own `GET /api/settings/css` endpoint.
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
pub struct SettingsDto { pub struct SettingsDto {
pub color_scheme: ColorScheme, pub color_scheme: ColorScheme,
pub active_profile: Option<String>, pub active_profile: Option<String>,
/// `None` means unlimited.
pub max_logs: Option<usize>, pub max_logs: Option<usize>,
/// Id of the selected custom theme entry, if any. `None` means `color_scheme` alone determines the active palette.
pub custom_theme_id: Option<String>, pub custom_theme_id: Option<String>,
/// Ntfy topic every alert rule delivers to; `None` until configured.
pub ntfy_topic: Option<String>, pub ntfy_topic: Option<String>,
/// Ntfy server override; `None` means `https://ntfy.sh`.
pub ntfy_server: Option<String>, pub ntfy_server: Option<String>,
} }
@@ -190,7 +167,7 @@ impl From<&Settings> for SettingsDto {
fn from(settings: &Settings) -> Self { fn from(settings: &Settings) -> Self {
SettingsDto { SettingsDto {
color_scheme: settings.color_scheme, color_scheme: settings.color_scheme,
active_profile: settings.profiles.active_profile.clone(), active_profile: settings.active_profile.clone(),
max_logs: settings.max_logs, max_logs: settings.max_logs,
custom_theme_id: settings.custom_theme_id.clone(), custom_theme_id: settings.custom_theme_id.clone(),
ntfy_topic: settings.ntfy_topic.clone(), ntfy_topic: settings.ntfy_topic.clone(),
@@ -199,7 +176,6 @@ impl From<&Settings> for SettingsDto {
} }
} }
/// `max_logs`/`custom_theme_id`/`ntfy_topic`/`ntfy_server` distinguish "omitted" (leave unchanged) from "present" (null clears, value sets), via `deserialize_double_option`.
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct UpdateSettingsRequest { pub struct UpdateSettingsRequest {
pub color_scheme: Option<ColorScheme>, pub color_scheme: Option<ColorScheme>,
@@ -213,7 +189,6 @@ pub struct UpdateSettingsRequest {
pub ntfy_server: Option<Option<String>>, pub ntfy_server: Option<Option<String>>,
} }
/// Distinguishes a missing field (outer `None`) from present `null` (`Some(None)`) or present value (`Some(Some(value))`), which plain `Option<Option<T>>` can't do alone.
fn deserialize_double_option<'de, D, T>(deserializer: D) -> Result<Option<Option<T>>, D::Error> fn deserialize_double_option<'de, D, T>(deserializer: D) -> Result<Option<Option<T>>, D::Error>
where where
D: serde::Deserializer<'de>, D: serde::Deserializer<'de>,
@@ -236,7 +211,6 @@ pub struct UpdateAlertRuleRequest {
pub condition: AlertCondition, pub condition: AlertCondition,
} }
/// `server` defaults to `https://ntfy.sh` when absent.
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct TestAlertNotificationRequest { pub struct TestAlertNotificationRequest {
pub topic: String, pub topic: String,
@@ -267,7 +241,6 @@ pub struct CreateProfileRequest {
pub name: String, pub name: String,
} }
/// Mirrors `profile::Panel`.
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PanelDto { pub struct PanelDto {
pub id: String, pub id: String,
@@ -298,7 +271,6 @@ impl From<&PanelDto> for Panel {
} }
} }
/// Mirrors `profile::Dashboard`.
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DashboardDto { pub struct DashboardDto {
pub id: String, pub id: String,
@@ -331,7 +303,6 @@ pub struct CreateDashboardRequest {
pub name: String, pub name: String,
} }
/// The dashboard `id` in the URL is authoritative; any `id` in the body is ignored.
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct SaveDashboardRequest { pub struct SaveDashboardRequest {
pub name: String, pub name: String,
src/api/mcp_server.rs
+971
View File
@@ -0,0 +1,971 @@
use std::collections::BTreeMap;
use rmcp::handler::server::router::tool::ToolRouter;
use rmcp::handler::server::wrapper::Parameters;
use rmcp::model::{
CallToolResult, ContentBlock, Implementation, ProtocolVersion, ServerCapabilities, ServerInfo,
};
use rmcp::{ErrorData as McpError, ServerHandler, schemars, tool, tool_handler, tool_router};
use serde::{Deserialize, Serialize};
use serde_json::Value;
// ---- logs / timeline / stats ----
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct QueryLogsParams {
#[serde(default)]
pub filter: Option<String>,
#[serde(default)]
pub count_only: bool,
#[serde(default)]
pub limit: Option<usize>,
#[serde(default)]
pub since_sequence: Option<u64>,
#[serde(default)]
pub start_ms: Option<i64>,
#[serde(default)]
pub end_ms: Option<i64>,
#[serde(default)]
pub sort_by: Option<String>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct GetTimelineParams {
#[serde(default)]
pub filter: Option<String>,
#[serde(default)]
pub start_ms: Option<i64>,
#[serde(default)]
pub end_ms: Option<i64>,
#[serde(default)]
pub bucket_ms: Option<i64>,
}
// ---- params ----
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct DeclareParamParams {
pub name: String,
pub message_match_regex: String,
pub capture_group: String,
#[serde(default)]
pub json_field_path: Option<String>,
pub kind: String,
#[serde(default)]
pub elapsed_since_last: bool,
}
// ---- metrics ----
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct DeclareMetricParams {
pub name: String,
pub message_match_regex: String,
#[serde(default)]
pub label_params: Vec<String>,
#[serde(default)]
pub value_param: Option<String>,
#[serde(default)]
pub matched_labels: Option<BTreeMap<String, String>>,
#[serde(default)]
pub unmatched_labels: Option<BTreeMap<String, String>>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct UpdateMetricParams {
/// Existing metric name to update.
pub name: String,
#[serde(flatten)]
pub declaration: DeclareMetricParams,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct DeleteMetricParams {
pub name: String,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct GetMetricBucketValuesParams {
pub name: String,
/// Comma-separated `label=value` pairs to filter buckets by, e.g. "status_code=500".
#[serde(default)]
pub labels: Option<String>,
#[serde(default)]
pub since_sequence: Option<u64>,
#[serde(default)]
pub start_ms: Option<i64>,
#[serde(default)]
pub end_ms: Option<i64>,
#[serde(default)]
pub limit: Option<usize>,
}
// ---- promql / dashboards ----
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct RunPromqlQueryParams {
pub promql: String,
#[serde(default)]
pub start_ms: Option<i64>,
#[serde(default)]
pub end_ms: Option<i64>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct CreateDashboardParams {
#[serde(default)]
pub profile: Option<String>,
pub name: String,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct AddPanelParams {
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
#[serde(default)]
pub panel_id: Option<String>,
pub title: String,
pub promql: String,
#[serde(default = "default_chart_type")]
pub chart_type: String,
}
fn default_chart_type() -> String {
"line".to_string()
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct GetDashboardParams {
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct ListDashboardsParams {
#[serde(default)]
pub profile: Option<String>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct DeletePanelParams {
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
pub panel_id: String,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct RenderPanelParams {
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
pub panel_id: String,
#[serde(default)]
pub start_ms: Option<i64>,
#[serde(default)]
pub end_ms: Option<i64>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct RenderDashboardParams {
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
#[serde(default)]
pub start_ms: Option<i64>,
#[serde(default)]
pub end_ms: Option<i64>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct DeleteDashboardParams {
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
}
// ---- settings / css ----
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct UpdateSettingsParams {
#[serde(default)]
pub color_scheme: Option<String>,
#[serde(default)]
pub max_logs: Option<Option<usize>>,
#[serde(default)]
pub custom_theme_id: Option<Option<String>>,
#[serde(default)]
pub ntfy_topic: Option<Option<String>>,
#[serde(default)]
pub ntfy_server: Option<Option<String>>,
}
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct ImportCssParams {
pub filename: String,
pub content: String,
}
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct LinkCssParams {
pub path_or_url: String,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct CssIdParams {
pub id: String,
}
// ---- alerts ----
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct CreateAlertRuleParams {
pub name: String,
pub enabled: bool,
/// One of: {"kind":"immediate","query":"..."}, {"kind":"threshold","query":"...","op":"gt|lt|ge|le|eq|ne","threshold":N,"window_ms":N}, {"kind":"newvalue","field_regex":"...","capture_group":"..."}, {"kind":"ratio","query_a":"...","query_b":"...","op":"...","ratio":N,"window_ms":N}
pub condition: Value,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct UpdateAlertRuleParams {
pub id: String,
pub name: String,
pub enabled: bool,
pub condition: Value,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct DeleteAlertRuleParams {
pub id: String,
}
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct TestAlertNotificationParams {
pub topic: String,
#[serde(default)]
pub server: Option<String>,
}
// ---- profiles ----
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct CreateProfileParams {
pub name: String,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct ProfileNameParams {
pub name: String,
}
#[derive(Clone)]
pub struct OttylMcp {
client: reqwest::Client,
base_url: String,
#[allow(dead_code)]
tool_router: ToolRouter<OttylMcp>,
}
fn mcp_error(context: &str, err: impl std::fmt::Display) -> McpError {
McpError::internal_error(format!("{context}: {err}"), None)
}
fn json_result(value: Value) -> Result<CallToolResult, McpError> {
Ok(CallToolResult::success(vec![ContentBlock::text(
serde_json::to_string_pretty(&value).unwrap_or_else(|_| value.to_string()),
)]))
}
#[tool_router]
impl OttylMcp {
pub fn new(base_url: String) -> Self {
Self {
client: reqwest::Client::new(),
base_url,
tool_router: Self::tool_router(),
}
}
fn url(&self, path: &str) -> String {
format!("{}{path}", self.base_url)
}
async fn resolve_profile(&self, explicit: Option<String>) -> Result<String, McpError> {
if let Some(profile) = explicit {
return Ok(profile);
}
let settings: Value = self
.client
.get(self.url("/api/settings"))
.send()
.await
.map_err(|err| mcp_error("failed to reach ottyl", err))?
.json()
.await
.map_err(|err| mcp_error("failed to parse settings response", err))?;
settings
.get("active_profile")
.and_then(Value::as_str)
.map(str::to_string)
.ok_or_else(|| {
McpError::invalid_params(
"no active profile on the target instance; pass `profile` explicitly",
None,
)
})
}
async fn get_json(&self, path: &str, query: &[(&str, String)]) -> Result<Value, McpError> {
let response = self
.client
.get(self.url(path))
.query(query)
.send()
.await
.map_err(|err| mcp_error("failed to reach ottyl", err))?;
let status = response.status();
let body: Value = response
.json()
.await
.map_err(|err| mcp_error("failed to parse response", err))?;
if !status.is_success() {
return Err(mcp_error("ottyl rejected the request", body));
}
Ok(body)
}
async fn send_json(
&self,
method: reqwest::Method,
path: &str,
body: Option<&Value>,
) -> Result<Value, McpError> {
let mut request = self.client.request(method, self.url(path));
if let Some(body) = body {
request = request.json(body);
}
let response = request
.send()
.await
.map_err(|err| mcp_error("failed to reach ottyl", err))?;
let status = response.status();
if status == reqwest::StatusCode::NO_CONTENT {
return Ok(Value::Null);
}
let body: Value = response
.json()
.await
.map_err(|err| mcp_error("failed to parse response", err))?;
if !status.is_success() {
return Err(mcp_error("ottyl rejected the request", body));
}
Ok(body)
}
async fn fetch_dashboard(&self, profile: &str, dashboard_id: &str) -> Result<Value, McpError> {
self.get_json(
&format!(
"/api/settings/profiles/{}/dashboards/{}",
urlencoding_path(profile),
urlencoding_path(dashboard_id)
),
&[],
)
.await
}
async fn run_promql(
&self,
promql: &str,
start_ms: Option<i64>,
end_ms: Option<i64>,
) -> Result<Value, McpError> {
let mut query: Vec<(&str, String)> = vec![("promql", promql.to_string())];
if let Some(start_ms) = start_ms {
query.push(("start_ms", start_ms.to_string()));
}
if let Some(end_ms) = end_ms {
query.push(("end_ms", end_ms.to_string()));
}
self.get_json("/api/query", &query).await
}
// ---- logs / timeline / stats ----
#[tool(description = "Count or read log lines matching a FilterBar query against the running ottyl instance.")]
async fn query_logs(
&self,
Parameters(params): Parameters<QueryLogsParams>,
) -> Result<CallToolResult, McpError> {
let mut query: Vec<(&str, String)> = Vec::new();
if let Some(filter) = &params.filter {
query.push(("filter", filter.clone()));
}
if let Some(limit) = params.limit {
query.push(("limit", limit.to_string()));
}
if let Some(since_sequence) = params.since_sequence {
query.push(("since_sequence", since_sequence.to_string()));
}
if let Some(start_ms) = params.start_ms {
query.push(("start_ms", start_ms.to_string()));
}
if let Some(end_ms) = params.end_ms {
query.push(("end_ms", end_ms.to_string()));
}
if let Some(sort_by) = &params.sort_by {
query.push(("sort_by", sort_by.clone()));
}
let logs = self.get_json("/api/logs", &query).await?;
if params.count_only {
let count = logs.as_array().map(Vec::len).unwrap_or(0);
json_result(serde_json::json!({ "count": count }))
} else {
json_result(logs)
}
}
#[tool(description = "Read the log-count timeline (bucketed histogram) matching an optional FilterBar query — the same data backing the Log Timeline chart.")]
async fn get_timeline(
&self,
Parameters(params): Parameters<GetTimelineParams>,
) -> Result<CallToolResult, McpError> {
let mut query: Vec<(&str, String)> = Vec::new();
if let Some(filter) = &params.filter {
query.push(("filter", filter.clone()));
}
if let Some(start_ms) = params.start_ms {
query.push(("start_ms", start_ms.to_string()));
}
if let Some(end_ms) = params.end_ms {
query.push(("end_ms", end_ms.to_string()));
}
if let Some(bucket_ms) = params.bucket_ms {
query.push(("bucket_ms", bucket_ms.to_string()));
}
json_result(self.get_json("/api/timeline", &query).await?)
}
#[tool(description = "Read overall ingest stats: log count, sequence/timestamp bounds, ingest lag.")]
async fn get_stats(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/stats", &[]).await?)
}
// ---- params ----
#[tool(description = "List all currently declared params (name, regex, kind, etc.), including auto-derived ones.")]
async fn list_params(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/params", &[]).await?)
}
#[tool(description = "Declare a param: a named value extracted from matching log lines.")]
async fn declare_param(
&self,
Parameters(params): Parameters<DeclareParamParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/params", Some(&body)).await?)
}
#[tool(description = "Clear all declared params and re-derive auto params from retained log lines.")]
async fn clear_params(&self) -> Result<CallToolResult, McpError> {
json_result(self.send_json(reqwest::Method::DELETE, "/api/params", None).await?)
}
// ---- metrics ----
#[tool(description = "List all currently declared metrics.")]
async fn list_metrics(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/metrics", &[]).await?)
}
#[tool(description = "Declare a metric: named event counts/values bucketed by label, derived from matching log lines.")]
async fn declare_metric(
&self,
Parameters(params): Parameters<DeclareMetricParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/metrics", Some(&body)).await?)
}
#[tool(description = "Update an existing metric declaration in place by name.")]
async fn update_metric(
&self,
Parameters(params): Parameters<UpdateMetricParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params.declaration)
.map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(
reqwest::Method::PUT,
&format!("/api/metrics/{}", urlencoding_path(&params.name)),
Some(&body),
)
.await?,
)
}
#[tool(description = "Delete a declared metric by name.")]
async fn delete_metric(
&self,
Parameters(params): Parameters<DeleteMetricParams>,
) -> Result<CallToolResult, McpError> {
json_result(
self.send_json(
reqwest::Method::DELETE,
&format!("/api/metrics/{}", urlencoding_path(&params.name)),
None,
)
.await?,
)
}
#[tool(description = "Read a metric's raw bucket values (labels, sequence, timestamp, value), optionally filtered by label and time range.")]
async fn get_metric_bucket_values(
&self,
Parameters(params): Parameters<GetMetricBucketValuesParams>,
) -> Result<CallToolResult, McpError> {
let mut query: Vec<(&str, String)> = Vec::new();
if let Some(labels) = &params.labels {
query.push(("labels", labels.clone()));
}
if let Some(since_sequence) = params.since_sequence {
query.push(("since_sequence", since_sequence.to_string()));
}
if let Some(start_ms) = params.start_ms {
query.push(("start_ms", start_ms.to_string()));
}
if let Some(end_ms) = params.end_ms {
query.push(("end_ms", end_ms.to_string()));
}
if let Some(limit) = params.limit {
query.push(("limit", limit.to_string()));
}
json_result(
self.get_json(&format!("/api/metrics/{}", urlencoding_path(&params.name)), &query)
.await?,
)
}
// ---- promql ----
#[tool(description = "Evaluate a PromQL-style query directly and get back raw series data (labels + timestamped points) — the same evaluation dashboards use to render charts.")]
async fn run_promql_query(
&self,
Parameters(params): Parameters<RunPromqlQueryParams>,
) -> Result<CallToolResult, McpError> {
json_result(self.run_promql(&params.promql, params.start_ms, params.end_ms).await?)
}
// ---- dashboards / panels ----
#[tool(description = "List dashboards saved in a profile (defaults to the active profile).")]
async fn list_dashboards(
&self,
Parameters(params): Parameters<ListDashboardsParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
json_result(
self.get_json(
&format!("/api/settings/profiles/{}/dashboards", urlencoding_path(&profile)),
&[],
)
.await?,
)
}
#[tool(description = "Create a new, empty dashboard in a profile.")]
async fn create_dashboard(
&self,
Parameters(params): Parameters<CreateDashboardParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
json_result(
self.send_json(
reqwest::Method::POST,
&format!("/api/settings/profiles/{}/dashboards", urlencoding_path(&profile)),
Some(&serde_json::json!({ "name": params.name })),
)
.await?,
)
}
#[tool(description = "Read a dashboard's current panels (definitions only — use render_panel or render_dashboard for the data a panel would chart).")]
async fn get_dashboard(
&self,
Parameters(params): Parameters<GetDashboardParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
json_result(self.fetch_dashboard(&profile, &params.dashboard_id).await?)
}
#[tool(description = "Delete a dashboard from a profile.")]
async fn delete_dashboard(
&self,
Parameters(params): Parameters<DeleteDashboardParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
json_result(
self.send_json(
reqwest::Method::DELETE,
&format!(
"/api/settings/profiles/{}/dashboards/{}",
urlencoding_path(&profile),
urlencoding_path(&params.dashboard_id)
),
None,
)
.await?,
)
}
#[tool(description = "Add a new panel to a dashboard, or update an existing one by panel_id.")]
async fn add_panel(
&self,
Parameters(params): Parameters<AddPanelParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
let mut dashboard = self.fetch_dashboard(&profile, &params.dashboard_id).await?;
let panel_id = params
.panel_id
.clone()
.unwrap_or_else(|| generate_panel_id(&params.title, &params.promql));
let new_panel = serde_json::json!({
"id": panel_id,
"title": params.title,
"promql": params.promql,
"chart_type": params.chart_type,
});
let panels = dashboard
.get_mut("panels")
.and_then(Value::as_array_mut)
.ok_or_else(|| mcp_error("unexpected dashboard shape", "missing panels array"))?;
match panels.iter().position(|panel| panel.get("id") == new_panel.get("id")) {
Some(index) => panels[index] = new_panel,
None => panels.push(new_panel),
}
json_result(self.save_dashboard(&profile, &params.dashboard_id, dashboard).await?)
}
#[tool(description = "Remove a panel from a dashboard by panel_id.")]
async fn delete_panel(
&self,
Parameters(params): Parameters<DeletePanelParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
let mut dashboard = self.fetch_dashboard(&profile, &params.dashboard_id).await?;
let panels = dashboard
.get_mut("panels")
.and_then(Value::as_array_mut)
.ok_or_else(|| mcp_error("unexpected dashboard shape", "missing panels array"))?;
panels.retain(|panel| panel.get("id").and_then(Value::as_str) != Some(params.panel_id.as_str()));
json_result(self.save_dashboard(&profile, &params.dashboard_id, dashboard).await?)
}
async fn save_dashboard(
&self,
profile: &str,
dashboard_id: &str,
dashboard: Value,
) -> Result<Value, McpError> {
self.send_json(
reqwest::Method::PUT,
&format!(
"/api/settings/profiles/{}/dashboards/{}",
urlencoding_path(profile),
urlencoding_path(dashboard_id)
),
Some(&serde_json::json!({ "name": dashboard.get("name"), "panels": dashboard.get("panels") })),
)
.await
}
#[tool(description = "Render one panel: resolve its promql from the dashboard definition, evaluate it, and return the chart's underlying series data (labels + timestamped points) as JSON — this is how an agent should 'see' a chart without a browser.")]
async fn render_panel(
&self,
Parameters(params): Parameters<RenderPanelParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
let dashboard = self.fetch_dashboard(&profile, &params.dashboard_id).await?;
let panel = dashboard
.get("panels")
.and_then(Value::as_array)
.and_then(|panels| panels.iter().find(|panel| panel.get("id").and_then(Value::as_str) == Some(params.panel_id.as_str())))
.ok_or_else(|| McpError::invalid_params("no panel with that panel_id in the dashboard", None))?;
let promql = panel
.get("promql")
.and_then(Value::as_str)
.ok_or_else(|| mcp_error("unexpected panel shape", "missing promql"))?;
let series = self.run_promql(promql, params.start_ms, params.end_ms).await?;
json_result(serde_json::json!({
"panel": panel,
"series": series,
}))
}
#[tool(description = "Render every panel in a dashboard: evaluate each panel's promql and return all their chart data as JSON, so an agent can reason about the whole dashboard without a browser.")]
async fn render_dashboard(
&self,
Parameters(params): Parameters<RenderDashboardParams>,
) -> Result<CallToolResult, McpError> {
let profile = self.resolve_profile(params.profile).await?;
let dashboard = self.fetch_dashboard(&profile, &params.dashboard_id).await?;
let panels = dashboard
.get("panels")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
let mut rendered = Vec::with_capacity(panels.len());
for panel in panels {
let promql = panel.get("promql").and_then(Value::as_str).unwrap_or_default();
let series = self.run_promql(promql, params.start_ms, params.end_ms).await?;
rendered.push(serde_json::json!({ "panel": panel, "series": series }));
}
json_result(serde_json::json!({
"dashboard": { "id": dashboard.get("id"), "name": dashboard.get("name") },
"panels": rendered,
}))
}
// ---- settings / theming ----
#[tool(description = "Read the running instance's global settings (color scheme, active profile, max_logs, ntfy config).")]
async fn get_settings(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/settings", &[]).await?)
}
#[tool(description = "Update global settings. Only fields provided are changed; pass null for an Option field to clear it.")]
async fn update_settings(
&self,
Parameters(params): Parameters<UpdateSettingsParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::PATCH, "/api/settings", Some(&body)).await?)
}
#[tool(description = "List imported/linked custom CSS theme entries.")]
async fn list_css(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/settings/css", &[]).await?)
}
#[tool(description = "Import a custom CSS theme file by content.")]
async fn import_css(
&self,
Parameters(params): Parameters<ImportCssParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/settings/css", Some(&body)).await?)
}
#[tool(description = "Link an external CSS file or URL as a custom theme, instead of importing its content.")]
async fn link_css(
&self,
Parameters(params): Parameters<LinkCssParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/settings/css/link", Some(&body)).await?)
}
#[tool(description = "Remove a custom CSS entry by id.")]
async fn remove_css(
&self,
Parameters(params): Parameters<CssIdParams>,
) -> Result<CallToolResult, McpError> {
json_result(
self.send_json(
reqwest::Method::DELETE,
&format!("/api/settings/css/{}", urlencoding_path(&params.id)),
None,
)
.await?,
)
}
#[tool(description = "Read the raw content of an imported CSS entry by id.")]
async fn get_css_content(
&self,
Parameters(params): Parameters<CssIdParams>,
) -> Result<CallToolResult, McpError> {
let response = self
.client
.get(self.url(&format!("/api/settings/css/{}/content", urlencoding_path(&params.id))))
.send()
.await
.map_err(|err| mcp_error("failed to reach ottyl", err))?;
let status = response.status();
let text = response
.text()
.await
.map_err(|err| mcp_error("failed to read css content response", err))?;
if !status.is_success() {
return Err(mcp_error("ottyl rejected the request", text));
}
json_result(serde_json::json!({ "content": text }))
}
// ---- alerts ----
#[tool(description = "List configured alert rules.")]
async fn list_alert_rules(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/settings/alerts", &[]).await?)
}
#[tool(description = "Create an alert rule.")]
async fn create_alert_rule(
&self,
Parameters(params): Parameters<CreateAlertRuleParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::json!({
"name": params.name,
"enabled": params.enabled,
"condition": params.condition,
});
json_result(self.send_json(reqwest::Method::POST, "/api/settings/alerts", Some(&body)).await?)
}
#[tool(description = "Update an existing alert rule by id.")]
async fn update_alert_rule(
&self,
Parameters(params): Parameters<UpdateAlertRuleParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::json!({
"name": params.name,
"enabled": params.enabled,
"condition": params.condition,
});
json_result(
self.send_json(
reqwest::Method::PUT,
&format!("/api/settings/alerts/{}", urlencoding_path(&params.id)),
Some(&body),
)
.await?,
)
}
#[tool(description = "Delete an alert rule by id.")]
async fn delete_alert_rule(
&self,
Parameters(params): Parameters<DeleteAlertRuleParams>,
) -> Result<CallToolResult, McpError> {
json_result(
self.send_json(
reqwest::Method::DELETE,
&format!("/api/settings/alerts/{}", urlencoding_path(&params.id)),
None,
)
.await?,
)
}
#[tool(description = "Send a test ntfy notification to verify alert delivery is configured correctly.")]
async fn test_alert_notification(
&self,
Parameters(params): Parameters<TestAlertNotificationParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(reqwest::Method::POST, "/api/settings/alerts/test", Some(&body))
.await?,
)
}
// ---- profiles ----
#[tool(description = "List all saved profiles with summary counts (params, metrics, dashboards).")]
async fn list_profiles(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/settings/profiles", &[]).await?)
}
#[tool(description = "Create a new, empty profile.")]
async fn create_profile(
&self,
Parameters(params): Parameters<CreateProfileParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/settings/profiles", Some(&body)).await?)
}
#[tool(description = "Delete a profile by name.")]
async fn delete_profile(
&self,
Parameters(params): Parameters<ProfileNameParams>,
) -> Result<CallToolResult, McpError> {
json_result(
self.send_json(
reqwest::Method::DELETE,
&format!("/api/settings/profiles/{}", urlencoding_path(&params.name)),
None,
)
.await?,
)
}
#[tool(description = "Activate a profile: swaps in its param/metric declarations and dashboards as the running instance's live state.")]
async fn activate_profile(
&self,
Parameters(params): Parameters<ProfileNameParams>,
) -> Result<CallToolResult, McpError> {
json_result(
self.send_json(
reqwest::Method::POST,
&format!("/api/settings/profiles/{}/activate", urlencoding_path(&params.name)),
None,
)
.await?,
)
}
}
fn urlencoding_path(segment: &str) -> String {
segment
.bytes()
.map(|b| match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
(b as char).to_string()
}
_ => format!("%{b:02X}"),
})
.collect()
}
fn generate_panel_id(title: &str, promql: &str) -> String {
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
use std::time::{SystemTime, UNIX_EPOCH};
let mut hasher = DefaultHasher::new();
title.hash(&mut hasher);
promql.hash(&mut hasher);
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
nanos.hash(&mut hasher);
format!("panel-{:016x}", hasher.finish())
}
#[tool_handler]
impl ServerHandler for OttylMcp {
fn get_info(&self) -> ServerInfo {
ServerInfo::new(
ServerCapabilities::builder().enable_tools().build(),
)
.with_server_info(Implementation::from_build_env())
.with_protocol_version(ProtocolVersion::V_2024_11_05)
.with_instructions(
"Proxies to a running ottyl instance's HTTP API. Covers everything the web UI can do: \
logs/timeline/stats, param & metric declarations, PromQL queries, dashboards & panels \
(including render_panel/render_dashboard, which return a chart's underlying series data \
as JSON so an agent can reason about it without a browser), settings & custom CSS themes, \
alert rules, and profile management. See the embedded SKILL.md (GET /SKILL.md on the \
target instance) for examples."
.to_string(),
)
}
}
src/api.rs → src/api/mod.rs
+571 -195
View File
File diff suppressed because it is too large Load Diff
src/cli/mod.rs
+205
View File
@@ -0,0 +1,205 @@
pub mod sessions;
use std::path::PathBuf;
use std::sync::{Arc, LazyLock, Mutex};
use clap::{Parser, Subcommand};
use tokio::net::TcpListener;
use crate::config::Settings;
static SETTINGS_PATH_HELP: LazyLock<String> = LazyLock::new(|| {
format!(
"Path to the settings file. Defaults to {}.",
crate::config::default_settings_path()
.map(|path| path.display().to_string())
.unwrap_or_else(|| "a per-OS config location".to_string())
)
});
#[derive(Parser, Debug)]
struct Args {
#[command(subcommand)]
command: Option<Command>,
#[arg(long, default_value_t = 0)]
port: u16,
#[arg(long, help = SETTINGS_PATH_HELP.as_str())]
settings_path: Option<PathBuf>,
#[arg(long)]
file: Option<PathBuf>,
#[arg(long)]
metrics_url: Option<String>,
#[arg(long)]
name: Option<String>,
}
#[derive(Subcommand, Debug)]
enum Command {
Mcp {
#[arg(long)]
session: Option<String>,
#[arg(long)]
port: Option<u16>,
},
Sessions,
}
pub async fn run() {
let args = Args::parse();
let settings_path = args.settings_path.clone().unwrap_or_else(|| {
crate::config::default_settings_path()
.expect("could not determine a default settings path; pass --settings-path")
});
match args.command {
Some(Command::Mcp { session, port }) => {
run_mcp(&settings_path, session, port).await;
return;
}
Some(Command::Sessions) => {
print_sessions(&settings_path);
return;
}
None => {}
}
let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
let engine_tx = crate::engine::spawn(settings.clone(), crate::consts::ENGINE_CHANNEL_CAPACITY);
{
let max_logs = settings.lock().unwrap().max_logs;
let _ = engine_tx.tx.send(crate::engine::EngineMessage::SetMaxLogs(max_logs)).await;
}
{
let (active_profile_name, active_profile) = {
let mut settings_guard = settings.lock().unwrap();
settings_guard
.ensure_default_profile(&settings_path)
.expect("failed to ensure the Default profile exists");
let active_profile_name = settings_guard.active_profile.clone();
let active_profile = active_profile_name
.as_deref()
.and_then(|name| settings_guard.profile(name).cloned());
(active_profile_name, active_profile)
};
match (active_profile_name, active_profile) {
(Some(_), Some(active_profile)) => {
active_profile.load_into_engine(&engine_tx).await;
}
(Some(profile_name), None) => {
eprintln!(
"warning: active profile '{profile_name}' not found in {}; skipping",
settings_path.display()
);
}
(None, _) => {}
}
}
match args.file.clone() {
Some(path) => {
tokio::spawn(crate::engine::ingest::run_file(engine_tx.clone(), path));
}
None => {
tokio::spawn(crate::engine::ingest::run(engine_tx.clone()));
}
}
if let Some(metrics_url) = args.metrics_url.clone() {
tokio::spawn(crate::engine::ingest::run_metrics_poll(engine_tx.clone(), metrics_url));
}
let app = crate::api::router(engine_tx, settings, settings_path.clone());
let listener = TcpListener::bind((crate::consts::DEFAULT_BIND_HOST, args.port))
.await
.expect("failed to bind listener");
let bound_port = listener
.local_addr()
.expect("failed to read back bound address")
.port();
let source = match &args.file {
Some(path) => format!("tailing {}", path.display()),
None => "reading logs from stdin".to_string(),
};
let session_name = args.name.clone().unwrap_or_else(sessions::generate_name);
match sessions::register_session(&settings_path, &session_name, bound_port) {
Ok(_) => eprintln!(
"ottyl serving on http://localhost:{} ({}), session name: {}",
bound_port, source, session_name
),
Err(err) => eprintln!(
"ottyl serving on http://localhost:{} ({}); warning: failed to register session '{}': {}",
bound_port, source, session_name, err
),
}
axum::serve(listener, app)
.await
.expect("server error");
}
async fn run_mcp(settings_path: &std::path::Path, session: Option<String>, port: Option<u16>) {
use rmcp::ServiceExt;
let port = match (port, session) {
(Some(port), _) => port,
(None, Some(name)) => match sessions::resolve_session(settings_path, &name) {
Some(info) => info.port,
None => {
eprintln!("error: no running ottyl session named '{name}'");
std::process::exit(1);
}
},
(None, None) => {
eprintln!("error: `ottyl mcp` requires --session <name> or --port <port>");
std::process::exit(1);
}
};
let base_url = format!("http://localhost:{port}");
let service = crate::api::mcp_server::OttylMcp::new(base_url)
.serve(rmcp::transport::stdio())
.await
.expect("failed to start MCP server");
service.waiting().await.expect("MCP server error");
}
fn print_sessions(settings_path: &std::path::Path) {
let dir = sessions::sessions_dir(settings_path);
let sessions = sessions::list_sessions(&dir);
if sessions.is_empty() {
println!("no known ottyl sessions");
return;
}
for session in sessions {
println!("{}\tport={}\tpid={}", session.name, session.port, session.pid);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn settings_path_defaults_to_none() {
let args = Args::parse_from(["ottyl"]);
assert_eq!(args.settings_path, None);
}
#[test]
fn settings_path_flag_overrides_default() {
let args = Args::parse_from(["ottyl", "--settings-path", "/tmp/custom-settings.json"]);
assert_eq!(
args.settings_path,
Some(PathBuf::from("/tmp/custom-settings.json"))
);
}
}
src/cli/sessions.rs
+158
View File
@@ -0,0 +1,158 @@
use std::fs;
use std::io;
use std::path::{Path, PathBuf};
use std::process;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SessionInfo {
pub name: String,
pub port: u16,
pub pid: u32,
}
pub fn sessions_dir(settings_path: &Path) -> PathBuf {
settings_path
.parent()
.map(|parent| parent.join("sessions"))
.unwrap_or_else(|| PathBuf::from("sessions"))
}
fn session_file_path(dir: &Path, name: &str) -> PathBuf {
dir.join(format!("{name}.json"))
}
#[cfg(unix)]
fn is_pid_alive(pid: u32) -> bool {
process::Command::new("kill")
.args(["-0", &pid.to_string()])
.stdout(process::Stdio::null())
.stderr(process::Stdio::null())
.status()
.map(|status| status.success())
.unwrap_or(true)
}
#[cfg(not(unix))]
fn is_pid_alive(_pid: u32) -> bool {
true
}
pub fn cleanup_stale(dir: &Path) {
let Ok(entries) = fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let path = entry.path();
if path.extension().and_then(|ext| ext.to_str()) != Some("json") {
continue;
}
let Ok(contents) = fs::read_to_string(&path) else {
continue;
};
let Ok(info) = serde_json::from_str::<SessionInfo>(&contents) else {
continue;
};
if !is_pid_alive(info.pid) {
let _ = fs::remove_file(&path);
}
}
}
pub fn generate_name() -> String {
petname::petname(2, "-").unwrap_or_else(|| format!("session-{}", process::id()))
}
pub fn register_session(settings_path: &Path, name: &str, port: u16) -> io::Result<PathBuf> {
let dir = sessions_dir(settings_path);
fs::create_dir_all(&dir)?;
cleanup_stale(&dir);
let info = SessionInfo {
name: name.to_string(),
port,
pid: process::id(),
};
let path = session_file_path(&dir, name);
fs::write(&path, serde_json::to_string_pretty(&info)?)?;
Ok(path)
}
pub fn resolve_session(settings_path: &Path, name: &str) -> Option<SessionInfo> {
let dir = sessions_dir(settings_path);
let contents = fs::read_to_string(session_file_path(&dir, name)).ok()?;
let info: SessionInfo = serde_json::from_str(&contents).ok()?;
is_pid_alive(info.pid).then_some(info)
}
pub fn list_sessions(dir: &Path) -> Vec<SessionInfo> {
let Ok(entries) = fs::read_dir(dir) else {
return Vec::new();
};
let mut sessions: Vec<SessionInfo> = entries
.flatten()
.filter(|entry| entry.path().extension().and_then(|ext| ext.to_str()) == Some("json"))
.filter_map(|entry| fs::read_to_string(entry.path()).ok())
.filter_map(|contents| serde_json::from_str(&contents).ok())
.collect();
sessions.sort_by(|a, b| a.name.cmp(&b.name));
sessions
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn register_session_writes_correct_port() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let path = register_session(&settings_path, "foo", 4242).unwrap();
assert!(path.exists());
let info = resolve_session(&settings_path, "foo").unwrap();
assert_eq!(info.port, 4242);
assert_eq!(info.name, "foo");
assert_eq!(info.pid, process::id());
}
#[test]
fn resolve_session_returns_none_for_unknown_name() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
assert!(resolve_session(&settings_path, "does-not-exist").is_none());
}
#[test]
fn stale_session_does_not_block_reregistration_under_same_name() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let sessions_dir_path = sessions_dir(&settings_path);
fs::create_dir_all(&sessions_dir_path).unwrap();
let stale = SessionInfo { name: "foo".to_string(), port: 1111, pid: 999_999 };
fs::write(
session_file_path(&sessions_dir_path, "foo"),
serde_json::to_string(&stale).unwrap(),
)
.unwrap();
register_session(&settings_path, "foo", 5555).unwrap();
let info = resolve_session(&settings_path, "foo").unwrap();
assert_eq!(info.port, 5555);
assert_eq!(info.pid, process::id());
}
#[test]
fn list_sessions_returns_all_entries_sorted_by_name() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
register_session(&settings_path, "zeta", 1).unwrap();
register_session(&settings_path, "alpha", 2).unwrap();
let sessions = list_sessions(&sessions_dir(&settings_path));
let names: Vec<&str> = sessions.iter().map(|s| s.name.as_str()).collect();
assert_eq!(names, vec!["alpha", "zeta"]);
}
}
src/settings.rs → src/config/mod.rs
+207 -76
View File
@@ -1,4 +1,4 @@
//! Process-wide settings model: colour scheme, CSS entries, and profile pointers, persisted as JSON. pub mod profile;
use std::collections::hash_map::DefaultHasher; use std::collections::hash_map::DefaultHasher;
use std::fmt; use std::fmt;
@@ -12,17 +12,11 @@ use std::time::{SystemTime, UNIX_EPOCH};
use directories::ProjectDirs; use directories::ProjectDirs;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use crate::alerts::AlertRule; use crate::engine::alerts::AlertRule;
use crate::config::profile::Profile;
/// Shared, process-wide settings handle. Cheap to clone.
pub type SettingsHandle = Arc<Mutex<Settings>>; pub type SettingsHandle = Arc<Mutex<Settings>>;
/// User's preferred colour scheme for the webapp.
///
/// `Dark` accepts the legacy `"grey"` serde value (ticket 004: "grey" was a
/// near-duplicate of "dark" and has been removed as a selectable option) so a
/// settings file saved before that change still loads cleanly instead of
/// falling back to `System`.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")] #[serde(rename_all = "lowercase")]
pub enum ColorScheme { pub enum ColorScheme {
@@ -35,15 +29,22 @@ pub enum ColorScheme {
Yellow, Yellow,
Pink, Pink,
Orange, Orange,
#[serde(rename = "light_green")]
LightGreen,
#[serde(rename = "light_red")]
LightRed,
#[serde(rename = "light_blue")]
LightBlue,
#[serde(rename = "light_yellow")]
LightYellow,
#[serde(rename = "light_pink")]
LightPink,
#[serde(rename = "light_orange")]
LightOrange,
#[default] #[default]
System, System,
} }
/// A single CSS entry: content ottyl stores itself ("imported") or a reference to an external
/// path/URL ("linked", not deletable via the API). Every entry is always a selectable
/// colour-scheme theme (ticket 020 removed the "use as theme"/always-on-overlay distinction);
/// an entry persisted before that change with a stored `is_theme: false` still loads fine (the
/// field is simply ignored) and becomes selectable like any other entry.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")] #[serde(tag = "kind", rename_all = "lowercase")]
pub enum CssEntry { pub enum CssEntry {
@@ -51,71 +52,106 @@ pub enum CssEntry {
Linked { path_or_url: String }, Linked { path_or_url: String },
} }
/// Tracks known profile names and which one is active; the profile files themselves live elsewhere. #[derive(Debug, Clone, Deserialize, Default)]
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default)] struct LegacyProfilesPointer {
pub struct Profiles { #[serde(default)]
/// Names of known profiles. active_profile: Option<String>,
pub names: Vec<String>, #[serde(default)]
/// Name of the currently active profile, if any. #[allow(dead_code)]
pub active_profile: Option<String>, profiles_dir: Option<String>,
/// Directory (relative to the settings file's directory, or absolute)
/// where individual profile files are expected to live.
pub profiles_dir: String,
} }
/// The single, process-wide settings instance. Only `PartialEq` (not `Eq`): `alert_rules` carries `f64` fields, which aren't `Eq`. #[derive(Debug, Clone, Deserialize)]
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(untagged)]
enum ProfilesOnDisk {
Inline(Vec<Profile>),
Legacy(LegacyProfilesPointer),
}
impl Default for ProfilesOnDisk {
fn default() -> Self {
ProfilesOnDisk::Inline(Vec::new())
}
}
#[derive(Debug, Clone, Deserialize)]
struct SettingsOnDisk {
color_scheme: ColorScheme,
#[serde(default)]
css_entries: Vec<CssEntry>,
#[serde(default)]
custom_theme_id: Option<String>,
#[serde(default)]
profiles: ProfilesOnDisk,
#[serde(default)]
active_profile: Option<String>,
#[serde(default)]
max_logs: Option<usize>,
#[serde(default)]
alert_rules: Vec<AlertRule>,
#[serde(default)]
ntfy_topic: Option<String>,
#[serde(default)]
ntfy_server: Option<String>,
}
impl From<SettingsOnDisk> for Settings {
fn from(on_disk: SettingsOnDisk) -> Self {
let (profiles, legacy_active_profile) = match on_disk.profiles {
ProfilesOnDisk::Inline(profiles) => (profiles, None),
ProfilesOnDisk::Legacy(pointer) => (Vec::new(), pointer.active_profile),
};
Settings {
color_scheme: on_disk.color_scheme,
css_entries: on_disk.css_entries,
custom_theme_id: on_disk.custom_theme_id,
profiles,
active_profile: on_disk.active_profile.or(legacy_active_profile),
max_logs: on_disk.max_logs,
alert_rules: on_disk.alert_rules,
ntfy_topic: on_disk.ntfy_topic,
ntfy_server: on_disk.ntfy_server,
}
}
}
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
#[serde(from = "SettingsOnDisk")]
pub struct Settings { pub struct Settings {
pub color_scheme: ColorScheme, pub color_scheme: ColorScheme,
pub css_entries: Vec<CssEntry>, pub css_entries: Vec<CssEntry>,
/// Id of the selected custom theme entry, if any; `None` means `color_scheme` alone determines the active palette.
#[serde(default)] #[serde(default)]
pub custom_theme_id: Option<String>, pub custom_theme_id: Option<String>,
pub profiles: Profiles, pub profiles: Vec<Profile>,
/// Maximum number of ingested logs kept in memory; `None` means unlimited. pub active_profile: Option<String>,
#[serde(default)] #[serde(default)]
pub max_logs: Option<usize>, pub max_logs: Option<usize>,
/// Alert rule definitions.
#[serde(default)] #[serde(default)]
pub alert_rules: Vec<AlertRule>, pub alert_rules: Vec<AlertRule>,
/// Ntfy topic every alert rule delivers to; `None` until configured in Settings.
#[serde(default)] #[serde(default)]
pub ntfy_topic: Option<String>, pub ntfy_topic: Option<String>,
/// Ntfy server override; `None` means `https://ntfy.sh`.
#[serde(default)] #[serde(default)]
pub ntfy_server: Option<String>, pub ntfy_server: Option<String>,
} }
impl Default for Settings {
fn default() -> Self {
Settings {
color_scheme: ColorScheme::default(),
css_entries: Vec::new(),
custom_theme_id: None,
profiles: Profiles {
names: Vec::new(),
active_profile: None,
profiles_dir: "profiles".to_string(),
},
max_logs: None,
alert_rules: Vec::new(),
ntfy_topic: None,
ntfy_server: None,
}
}
}
/// Computes the default settings file path (`<config_dir>/settings.json`) from OS-conventional config directories; `None` if none could be determined.
pub fn default_settings_path() -> Option<PathBuf> { pub fn default_settings_path() -> Option<PathBuf> {
ProjectDirs::from("", "", "ottyl").map(|dirs| dirs.config_dir().join("settings.json")) ProjectDirs::from(
crate::consts::APP_QUALIFIER,
crate::consts::APP_ORGANIZATION,
crate::consts::APP_NAME,
)
.map(|dirs| dirs.config_dir().join(crate::consts::SETTINGS_FILE_NAME))
} }
impl Settings { impl Settings {
/// Loads settings from `path`, creating and persisting defaults if missing. Any I/O or parse error on an existing file is logged and treated as defaults, without overwriting the unreadable file.
pub fn load(path: &Path) -> Settings { pub fn load(path: &Path) -> Settings {
match fs::read_to_string(path) { match fs::read_to_string(path) {
Ok(contents) => match serde_json::from_str(&contents) { Ok(contents) => match serde_json::from_str::<Settings>(&contents) {
Ok(settings) => settings, Ok(mut settings) => {
settings.migrate_legacy_profiles_file(path);
settings.migrate_legacy_profile_shapes(path);
settings
}
Err(err) => { Err(err) => {
eprintln!( eprintln!(
"warning: failed to parse settings file at {}: {err}; using defaults", "warning: failed to parse settings file at {}: {err}; using defaults",
@@ -125,7 +161,8 @@ impl Settings {
} }
}, },
Err(err) if err.kind() == io::ErrorKind::NotFound => { Err(err) if err.kind() == io::ErrorKind::NotFound => {
let settings = Settings::default(); let mut settings = Settings::default();
settings.migrate_legacy_profiles_file(path);
if let Err(save_err) = settings.save(path) { if let Err(save_err) = settings.save(path) {
eprintln!( eprintln!(
"warning: failed to create default settings file at {}: {save_err}", "warning: failed to create default settings file at {}: {save_err}",
@@ -144,7 +181,6 @@ impl Settings {
} }
} }
/// Persists these settings to `path` as pretty-printed JSON, creating any missing parent directories first.
pub fn save(&self, path: &Path) -> io::Result<()> { pub fn save(&self, path: &Path) -> io::Result<()> {
if let Some(parent) = path.parent() { if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?; fs::create_dir_all(parent)?;
@@ -153,16 +189,52 @@ impl Settings {
.map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err))?; .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err))?;
fs::write(path, contents) fs::write(path, contents)
} }
fn migrate_legacy_profiles_file(&mut self, settings_path: &Path) {
if !self.profiles.is_empty() {
return;
}
let legacy_path = settings_path
.parent()
.map(|parent| parent.join("profiles"))
.unwrap_or_else(|| PathBuf::from("profiles"))
.join("profiles.json");
let Some(profiles) = crate::config::profile::Profile::load_legacy_store(&legacy_path) else {
return;
};
self.profiles = profiles;
if let Err(err) = self.save(settings_path) {
eprintln!(
"warning: failed to persist profiles migrated from {}: {err}",
legacy_path.display()
);
return;
}
let migrated_path = legacy_path.with_extension("json.migrated");
if let Err(err) = fs::rename(&legacy_path, &migrated_path) {
eprintln!(
"warning: migrated profiles from {} into settings.json, but failed to rename the old file: {err}",
legacy_path.display()
);
}
}
fn migrate_legacy_profile_shapes(&mut self, settings_path: &Path) {
if crate::config::profile::migrate_legacy_profile_shapes(&mut self.profiles)
&& let Err(err) = self.save(settings_path)
{
eprintln!(
"warning: failed to persist migrated profile shapes to {}: {err}",
settings_path.display()
);
}
}
} }
/// Error returned by CSS-entry mutations and lookups.
#[derive(Debug)] #[derive(Debug)]
pub enum CssError { pub enum CssError {
/// No entry with the given id exists.
NotFound, NotFound,
/// The entry is a `Linked` entry, not `Imported` — ottyl doesn't own a linked entry's content, so there's nothing to delete or serve back.
NotImported, NotImported,
/// Underlying I/O failure: writing/reading/removing the CSS file, or persisting `settings.json`.
Io(io::Error), Io(io::Error),
} }
@@ -184,7 +256,6 @@ impl From<io::Error> for CssError {
} }
} }
/// Computes the id used to reference a `CssEntry` over the API: the stored id for `Imported`, or a hash of `path_or_url` for `Linked` (which has no stored id), so the frontend can key both the same way.
pub fn css_entry_id(entry: &CssEntry) -> String { pub fn css_entry_id(entry: &CssEntry) -> String {
match entry { match entry {
CssEntry::Imported { id, .. } => id.clone(), CssEntry::Imported { id, .. } => id.clone(),
@@ -198,7 +269,6 @@ fn linked_entry_id(path_or_url: &str) -> String {
format!("linked-{:016x}", hasher.finish()) format!("linked-{:016x}", hasher.finish())
} }
/// Generates an id for a newly-imported CSS entry by hashing filename, content, and time, avoiding a `uuid` dependency for what's otherwise an opaque string.
fn generate_imported_css_id(filename: &str, content: &str) -> String { fn generate_imported_css_id(filename: &str, content: &str) -> String {
let mut hasher = DefaultHasher::new(); let mut hasher = DefaultHasher::new();
filename.hash(&mut hasher); filename.hash(&mut hasher);
@@ -216,7 +286,6 @@ fn imported_css_file_path(css_dir: &Path, id: &str) -> PathBuf {
} }
impl Settings { impl Settings {
/// Imports raw CSS `content`: writes it to `<css_dir>/<id>.css`, records a `CssEntry::Imported` in `css_entries`, and persists settings. Returns the generated id. `css_dir` is expected to be the settings file's parent directory.
pub fn import_css( pub fn import_css(
&mut self, &mut self,
settings_path: &Path, settings_path: &Path,
@@ -235,14 +304,12 @@ impl Settings {
Ok(id) Ok(id)
} }
/// Registers a reference to an external CSS path/URL without copying its content, and persists settings.
pub fn link_css(&mut self, settings_path: &Path, path_or_url: String) -> Result<(), CssError> { pub fn link_css(&mut self, settings_path: &Path, path_or_url: String) -> Result<(), CssError> {
self.css_entries.push(CssEntry::Linked { path_or_url }); self.css_entries.push(CssEntry::Linked { path_or_url });
self.save(settings_path)?; self.save(settings_path)?;
Ok(()) Ok(())
} }
/// Removes an imported CSS entry identified by `id`: deletes its file under `css_dir`, drops it from `css_entries`, and persists settings. Fails with `NotFound` or `NotImported` (linked entries aren't deletable this way).
pub fn remove_css( pub fn remove_css(
&mut self, &mut self,
settings_path: &Path, settings_path: &Path,
@@ -259,7 +326,6 @@ impl Settings {
Some(CssEntry::Imported { .. }) => { Some(CssEntry::Imported { .. }) => {
let idx = index.unwrap(); let idx = index.unwrap();
self.css_entries.remove(idx); self.css_entries.remove(idx);
// Best-effort: the settings entry is already removed even if the file is gone.
let _ = fs::remove_file(imported_css_file_path(css_dir, id)); let _ = fs::remove_file(imported_css_file_path(css_dir, id));
self.save(settings_path)?; self.save(settings_path)?;
Ok(()) Ok(())
@@ -267,7 +333,6 @@ impl Settings {
} }
} }
/// Reads back the raw content of an imported CSS entry identified by `id`. Fails with `NotFound` or `NotImported` (linked entries have no local content — the frontend fetches `path_or_url` directly).
pub fn css_content(&self, css_dir: &Path, id: &str) -> Result<String, CssError> { pub fn css_content(&self, css_dir: &Path, id: &str) -> Result<String, CssError> {
match self.css_entries.iter().find(|entry| css_entry_id(entry) == id) { match self.css_entries.iter().find(|entry| css_entry_id(entry) == id) {
None => Err(CssError::NotFound), None => Err(CssError::NotFound),
@@ -285,7 +350,6 @@ mod tests {
#[test] #[test]
fn default_settings_path_ends_with_settings_json() { fn default_settings_path_ends_with_settings_json() {
// Can't assert the exact OS path without touching real config dirs, so just check the shape.
if let Some(path) = default_settings_path() { if let Some(path) = default_settings_path() {
assert_eq!(path.file_name().unwrap(), "settings.json"); assert_eq!(path.file_name().unwrap(), "settings.json");
assert!(path.parent().is_some()); assert!(path.parent().is_some());
@@ -306,6 +370,23 @@ mod tests {
assert_eq!(reloaded, settings); assert_eq!(reloaded, settings);
} }
#[test]
fn light_variant_color_schemes_round_trip_through_serde() {
for (scheme, rename) in [
(ColorScheme::LightGreen, "light_green"),
(ColorScheme::LightRed, "light_red"),
(ColorScheme::LightBlue, "light_blue"),
(ColorScheme::LightYellow, "light_yellow"),
(ColorScheme::LightPink, "light_pink"),
(ColorScheme::LightOrange, "light_orange"),
] {
let json = serde_json::to_string(&scheme).unwrap();
assert_eq!(json, format!("\"{rename}\""));
let deserialized: ColorScheme = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized, scheme);
}
}
#[test] #[test]
fn legacy_grey_color_scheme_deserializes_as_dark() { fn legacy_grey_color_scheme_deserializes_as_dark() {
let settings: Settings = serde_json::from_str( let settings: Settings = serde_json::from_str(
@@ -315,6 +396,16 @@ mod tests {
assert_eq!(settings.color_scheme, ColorScheme::Dark); assert_eq!(settings.color_scheme, ColorScheme::Dark);
} }
#[test]
fn legacy_profiles_pointer_shape_deserializes_with_empty_inline_profiles() {
let settings: Settings = serde_json::from_str(
r#"{"color_scheme":"dark","css_entries":[],"custom_theme_id":null,"profiles":{"names":["web"],"active_profile":"web","profiles_dir":"profiles"},"max_logs":null,"alert_rules":[],"ntfy_topic":null,"ntfy_server":null}"#,
)
.unwrap();
assert!(settings.profiles.is_empty());
assert_eq!(settings.active_profile, Some("web".to_string()));
}
#[test] #[test]
fn round_trips_through_save_and_load() { fn round_trips_through_save_and_load() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
@@ -329,13 +420,13 @@ mod tests {
settings.css_entries.push(CssEntry::Linked { settings.css_entries.push(CssEntry::Linked {
path_or_url: "https://example.com/style.css".to_string(), path_or_url: "https://example.com/style.css".to_string(),
}); });
settings.profiles.names.push("default".to_string()); settings.profiles.push(crate::config::profile::Profile::new("default".to_string()));
settings.profiles.active_profile = Some("default".to_string()); settings.active_profile = Some("default".to_string());
settings.alert_rules.push(crate::alerts::AlertRule { settings.alert_rules.push(crate::engine::alerts::AlertRule {
id: "rule1".to_string(), id: "rule1".to_string(),
name: "error spike".to_string(), name: "error spike".to_string(),
enabled: true, enabled: true,
condition: crate::alerts::AlertCondition::Immediate { condition: crate::engine::alerts::AlertCondition::Immediate {
query: r#"regex("error")"#.to_string(), query: r#"regex("error")"#.to_string(),
}, },
}); });
@@ -378,7 +469,6 @@ mod tests {
assert!(css_file.exists()); assert!(css_file.exists());
assert_eq!(fs::read_to_string(&css_file).unwrap(), "body { color: red; }"); assert_eq!(fs::read_to_string(&css_file).unwrap(), "body { color: red; }");
// Drop and reload settings from disk (simulates a restart).
drop(settings); drop(settings);
let reloaded = Settings::load(&settings_path); let reloaded = Settings::load(&settings_path);
assert!(reloaded.css_entries.iter().any(|entry| matches!( assert!(reloaded.css_entries.iter().any(|entry| matches!(
@@ -428,7 +518,6 @@ mod tests {
let result = settings.remove_css(&settings_path, &css_dir, &linked_id); let result = settings.remove_css(&settings_path, &css_dir, &linked_id);
assert!(matches!(result, Err(CssError::NotImported))); assert!(matches!(result, Err(CssError::NotImported)));
// Entry must still be there — rejection shouldn't silently drop it.
assert_eq!(settings.css_entries.len(), 1); assert_eq!(settings.css_entries.len(), 1);
} }
@@ -458,4 +547,46 @@ mod tests {
let result = settings.css_content(&css_dir, &linked_id); let result = settings.css_content(&css_dir, &linked_id);
assert!(matches!(result, Err(CssError::NotImported))); assert!(matches!(result, Err(CssError::NotImported)));
} }
#[test]
fn load_migrates_legacy_profiles_file_into_inline_settings() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let profiles_dir = dir.path().join("profiles");
fs::create_dir_all(&profiles_dir).unwrap();
let legacy_path = profiles_dir.join("profiles.json");
fs::write(
&legacy_path,
r#"[{"name":"web","params":[],"metrics":[],"dashboards":[]}]"#,
)
.unwrap();
fs::write(
&settings_path,
r#"{"color_scheme":"system","css_entries":[],"custom_theme_id":null,"profiles":{"names":["web"],"active_profile":"web","profiles_dir":"profiles"},"max_logs":null,"alert_rules":[],"ntfy_topic":null,"ntfy_server":null}"#,
)
.unwrap();
let settings = Settings::load(&settings_path);
assert_eq!(settings.profiles.len(), 1);
assert_eq!(settings.profiles[0].name, "web");
assert_eq!(settings.active_profile, Some("web".to_string()));
assert!(!legacy_path.exists());
let migrated_path = legacy_path.with_extension("json.migrated");
assert!(migrated_path.exists());
let reloaded = Settings::load(&settings_path);
assert_eq!(reloaded.profiles.len(), 1);
assert_eq!(reloaded.profiles[0].name, "web");
}
#[test]
fn load_without_legacy_file_leaves_profiles_empty() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let settings = Settings::load(&settings_path);
assert!(settings.profiles.is_empty());
assert!(!dir.path().join("profiles").exists());
}
} }
src/config/profile.rs
+1357
View File
File diff suppressed because it is too large Load Diff
src/consts.rs
+54
View File
@@ -0,0 +1,54 @@
use std::time::Duration;
// mpsc channel capacity between ingest/API callers and the engine actor task.
pub const ENGINE_CHANNEL_CAPACITY: usize = 1024;
// Hard ceiling on how many params `auto_declare_params` will ever create automatically over the
// life of the engine, so one unusual input can't inflate the declared-param set forever.
pub const MAX_AUTO_DECLARED_PARAMS: usize = 64;
// Per-line ceiling on how many *new* params a single ingested line may contribute to the
// auto-derived set, so one unusually wide line can't burn the whole per-run budget at once.
pub const MAX_NEW_AUTO_PARAMS_PER_LINE: usize = 8;
// How long a sort helper index may sit unused before it's purged and left to rebuild on demand.
pub const SORT_HELPER_INDEX_PURGE_AFTER: Duration = Duration::from_secs(300);
// Window and capacity for the rolling ingestion-to-engine-processing lag high watermark.
pub const INGEST_LAG_WATERMARK_WINDOW_MS: i64 = 5000;
pub const INGEST_LAG_WATERMARK_MAX_ENTRIES: usize = 5;
// Bucket width used to merge multiple sub-series into one combined time series.
pub const MERGE_WINDOW_MS: i64 = 1_000;
// Default timeline query bucket width when the caller doesn't specify one.
pub const DEFAULT_TIMELINE_BUCKET_MS: i64 = 1_000;
// Reserved param names usable without declaring anything (ottyl-synthesized columns).
pub const OTTYL_RESERVED_PARAM_NAMES: [&str; 3] = ["ottyl_seq", "ottyl_ts", "ottyl_fmt"];
// Idle poll cadence while tailing stdin/a file for new log lines.
pub const STDIN_POLL_INTERVAL: Duration = Duration::from_millis(200);
// Poll cadence for a configured Prometheus/OpenMetrics `--metrics-url`.
pub const METRICS_POLL_INTERVAL: Duration = Duration::from_secs(15);
// Column-splitting regex fragments for the fixed-width logfmt column parser.
pub const COLUMN_SEPARATOR_PATTERN: &str = r"(?:\t|[ \t]{2,})";
pub const COLUMN_VALUE_PATTERN: &str = r"\S+(?: \S+)*";
// Name of the always-present profile every Settings file is seeded with.
pub const DEFAULT_PROFILE_NAME: &str = "Default";
// Name/label of the metric the Default profile is seeded with.
pub const SEEDED_METRIC_NAME: &str = "log_category";
pub const SEEDED_METRIC_LABEL: &str = "category";
// Host the HTTP server binds to; port is user-configurable, host is not.
pub const DEFAULT_BIND_HOST: &str = "0.0.0.0";
// App identity used to resolve the OS-conventional settings directory, and the settings file name.
pub const APP_QUALIFIER: &str = "";
pub const APP_ORGANIZATION: &str = "";
pub const APP_NAME: &str = "ottyl";
pub const SETTINGS_FILE_NAME: &str = "settings.json";
src/data_value.rs
-106
View File
@@ -1,106 +0,0 @@
use crate::metric_primitive::MetricPrimitive;
use crate::timestamp::Timestamp;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RegexToMetricPrimitiveMatcher {
pub param_name: String,
pub capture_group: String,
/// Alternative extraction source to `capture_group`: a dot-separated JSON field path (e.g.
/// `foo.bar`), read from lines that parse as a JSON object instead of via the regex capture
/// group. When present and non-empty, this takes precedence over `capture_group`.
/// `#[serde(default)]` so profile files/requests saved before this field existed still load
/// as plain regex-capture-group matchers.
#[serde(default)]
pub json_field_path: Option<String>,
pub kind: MetricPrimitiveKind,
/// Whether this param is a bucket label or its captured value (at most one Value-role param per metric); `#[serde(default)]` so profile files saved before this field existed still load as `Label`.
#[serde(default)]
pub role: ParamRole,
}
/// Reads a value out of a JSON object by dot-separated field path (e.g. `"foo.bar"`), for
/// [`RegexToMetricPrimitiveMatcher::json_field_path`] and the sibling `json_field_path` on a
/// standalone param declaration. Returns `None` when `text` doesn't parse as a JSON object, the
/// path doesn't resolve to a value, or the resolved value isn't a scalar (string/number/bool) —
/// there's no sensible flat text form of a nested object/array to feed into
/// `parse_metric_primitive`.
pub fn extract_json_field(text: &str, field_path: &str) -> Option<String> {
let root: serde_json::Value = serde_json::from_str(text).ok()?;
let mut current = root.as_object()?;
let mut segments = field_path.split('.').peekable();
loop {
let segment = segments.next()?;
let next_value = current.get(segment)?;
if segments.peek().is_none() {
return match next_value {
serde_json::Value::String(s) => Some(s.clone()),
serde_json::Value::Number(n) => Some(n.to_string()),
serde_json::Value::Bool(b) => Some(b.to_string()),
_ => None,
};
}
current = next_value.as_object()?;
}
}
/// See [`RegexToMetricPrimitiveMatcher::role`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")]
pub enum ParamRole {
#[default]
Label,
Value,
}
/// `rename_all = "lowercase"` matches other `kind`-style enums, so JSON bodies use "string"/"number"/"timestamp".
#[allow(dead_code)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MetricPrimitiveKind {
String,
Number,
Timestamp,
/// On a Value-role param this is derived as elapsed time since the previous event (not parsed from text); otherwise it's a literal i64 millisecond count.
Duration,
/// A standalone param declared with this kind doesn't require (or use) a capture group: its value is
/// simply whether `message_match_regex` matches the line, case-insensitively or per the regex's own
/// flags — see `crate::engine::derive_log_params`. Unlike other kinds it's present as `true`/`false`
/// on every line rather than omitted when unmatched.
Boolean,
}
/// Value type paired with [`crate::index_key::IndexKey`] in the engine's primary index.
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize)]
pub enum DataValue {
/// Paired with `IndexKey::IngestedLog`.
LogLiteral { text: String },
/// Paired with `IndexKey::ParamDeclaration`.
ParamDeclaration {
message_match_regex: String,
capture_group: String,
/// See `RegexToMetricPrimitiveMatcher::json_field_path`.
json_field_path: Option<String>,
kind: MetricPrimitiveKind,
},
/// Paired with `IndexKey::MetricDeclaration`.
MetricDeclaration {
message_match_regex: String,
params: Vec<RegexToMetricPrimitiveMatcher>,
/// See `crate::engine::DeclareMetricRequest::matched_labels`.
matched_labels: Option<BTreeMap<String, String>>,
/// See `crate::engine::DeclareMetricRequest::unmatched_labels`.
unmatched_labels: Option<BTreeMap<String, String>>,
},
/// Paired with `IndexKey::Bucket`.
BucketValue {
values: Vec<(u64, Timestamp, MetricPrimitive)>,
},
/// Paired with `IndexKey::LastMetricEvent`.
LastEventTimestamp {
timestamp: Timestamp,
},
}
src/engine.rs
-2621
View File
File diff suppressed because it is too large Load Diff
src/alerts.rs → src/engine/alerts.rs
+109 -77
View File
@@ -1,9 +1,5 @@
//! Alert rules: definitions, trigger evaluation, and ntfy delivery.
//!
//! Alert queries are restricted to regex/sequence/timestamp/json[field] — a metric[param] comparison would require a full bucket rescan per line, breaking the proportional-cost guarantee. json[field] is fine here since, like regex/sequence/timestamp, it only inspects the line's own text/fields, no index lookup.
use std::collections::hash_map::DefaultHasher; use std::collections::hash_map::DefaultHasher;
use std::collections::{HashMap, HashSet, VecDeque}; use std::collections::{HashMap, HashSet};
use std::fmt; use std::fmt;
use std::hash::{Hash, Hasher}; use std::hash::{Hash, Hasher};
use std::io; use std::io;
@@ -13,11 +9,12 @@ use std::time::{SystemTime, UNIX_EPOCH};
use regex::Regex; use regex::Regex;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use crate::engine::{apply_compare_op, eval_json_comparison}; use crate::engine::{self, eval_json_comparison};
use crate::filter_datetime; use crate::engine::filter_datetime;
use crate::filter_parser::{self, CompareOp, FilterExpr, FilterValue}; use crate::engine::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::settings::Settings; use crate::engine::rate::RateWindow;
use crate::timestamp::Timestamp; use crate::config::Settings;
use crate::engine::timestamp::Timestamp;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct AlertRule { pub struct AlertRule {
@@ -31,19 +28,16 @@ pub struct AlertRule {
#[serde(tag = "kind", rename_all = "lowercase")] #[serde(tag = "kind", rename_all = "lowercase")]
pub enum AlertCondition { pub enum AlertCondition {
Immediate { query: String }, Immediate { query: String },
/// Fires when `query`'s match count within a rolling `window_ms` crosses `threshold` (via `op`).
Threshold { Threshold {
query: String, query: String,
op: CompareOp, op: CompareOp,
threshold: f64, threshold: f64,
window_ms: i64, window_ms: i64,
}, },
/// Seen values live only in memory — a restart forgets them, so previously-seen values can fire again; a deliberate simplicity tradeoff, not an oversight.
NewValue { NewValue {
field_regex: String, field_regex: String,
capture_group: String, capture_group: String,
}, },
/// A zero `count_b` skips evaluation for that line rather than treating the ratio as infinite.
Ratio { Ratio {
query_a: String, query_a: String,
query_b: String, query_b: String,
@@ -76,7 +70,6 @@ impl From<io::Error> for AlertError {
} }
} }
/// Hashes the name and current time rather than pulling in a `uuid` dependency.
fn generate_alert_rule_id(name: &str) -> String { fn generate_alert_rule_id(name: &str) -> String {
let mut hasher = DefaultHasher::new(); let mut hasher = DefaultHasher::new();
name.hash(&mut hasher); name.hash(&mut hasher);
@@ -135,27 +128,34 @@ impl Settings {
} }
} }
/// Per-rule in-memory evaluation state; lives only for the process's lifetime — not persisted, not backfilled against already-ingested logs.
#[derive(Debug, Default)] #[derive(Debug, Default)]
pub struct AlertRuntimeState { pub struct AlertRuntimeState {
threshold_matches: VecDeque<i64>, threshold_window: RateWindow,
ratio_a_matches: VecDeque<i64>, threshold_backfilled: bool,
ratio_b_matches: VecDeque<i64>, ratio_a_window: RateWindow,
ratio_a_backfilled: bool,
ratio_b_window: RateWindow,
ratio_b_backfilled: bool,
seen_values: HashSet<String>, seen_values: HashSet<String>,
} }
/// `timestamps` is push_back-ordered, so once the front entry is within the window, every entry behind it is too. fn backfill_window(
fn prune_window(timestamps: &mut VecDeque<i64>, now_ms: i64, window_ms: i64) { snapshot: &engine::EngineSnapshot,
while let Some(&front) = timestamps.front() { query: &str,
if now_ms - front > window_ms { before_sequence: u64,
timestamps.pop_front(); now: Timestamp,
} else { window_ms: i64,
break; ) -> RateWindow {
} let since_ms = now.as_millis() - window_ms;
} let matching_timestamps =
snapshot.logs_since(before_sequence, since_ms, now.as_millis()).filter_map(
|(sequence, timestamp, text)| {
(eval_alert_query(query, sequence, timestamp, text) == Ok(true)).then_some(timestamp)
},
);
RateWindow::backfill(matching_timestamps, now, window_ms)
} }
/// Evaluates a FilterBar query against a single line's own fields only (metric[param] comparisons aren't supported — see module doc; json[field] is supported since it's also line-local).
fn eval_alert_query(query: &str, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> { fn eval_alert_query(query: &str, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
let expr = filter_parser::parse(query).map_err(|err| err.to_string())?; let expr = filter_parser::parse(query).map_err(|err| err.to_string())?;
eval_alert_expr(&expr, sequence, timestamp, text) eval_alert_expr(&expr, sequence, timestamp, text)
@@ -167,26 +167,33 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
let re = Regex::new(pattern).map_err(|err| format!("invalid regex in alert query: {err}"))?; let re = Regex::new(pattern).map_err(|err| format!("invalid regex in alert query: {err}"))?;
Ok(re.is_match(text)) Ok(re.is_match(text))
} }
FilterExpr::Comparison { metric, param: None, op, value } if metric == "sequence" => { FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_seq" => {
let target = match value { let target = match value {
FilterValue::Number(n) => *n, FilterValue::Number(n) => *n,
FilterValue::Raw(raw) => raw FilterValue::Raw(raw) => raw
.parse::<f64>() .parse::<f64>()
.map_err(|_| format!("'sequence' comparisons require a numeric value, got {raw:?}"))?, .map_err(|_| format!("'ottyl_seq' comparisons require a numeric value, got {raw:?}"))?,
}; };
Ok(apply_compare_op((sequence as f64).total_cmp(&target), *op)) Ok(op.matches((sequence as f64).total_cmp(&target)))
} }
FilterExpr::Comparison { metric, param: None, op, value } if metric == "timestamp" => { FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_ts" => {
let target_ms = match value { let target_ms = match value {
FilterValue::Number(n) => *n as i64, FilterValue::Number(n) => *n as i64,
FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, Timestamp::now()) FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, Timestamp::now())
.map_err(|err| err.to_string())? .map_err(|err| err.to_string())?
.as_millis(), .as_millis(),
}; };
Ok(apply_compare_op(timestamp.as_millis().cmp(&target_ms), *op)) Ok(op.matches(timestamp.as_millis().cmp(&target_ms)))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_fmt" => {
let target = match value {
FilterValue::Number(n) => n.to_string(),
FilterValue::Raw(raw) => raw.clone(),
};
Ok(op.matches(crate::engine::logfmt::detect_format(text).cmp(target.as_str())))
} }
FilterExpr::Comparison { metric, .. } => Err(format!( FilterExpr::Comparison { metric, .. } => Err(format!(
"alert queries only support regex()/sequence/timestamp (not metric comparisons like \ "alert queries only support regex()/ottyl_seq/ottyl_ts/ottyl_fmt (not metric comparisons like \
'{metric}[param]'): metric-based alert conditions aren't supported yet" '{metric}[param]'): metric-based alert conditions aren't supported yet"
)), )),
FilterExpr::JsonComparison { field, op, value } => { FilterExpr::JsonComparison { field, op, value } => {
@@ -201,10 +208,10 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
} }
} }
/// A malformed query (bad regex, unsupported metric comparison) is logged and simply never matches, rather than panicking.
pub fn evaluate_alerts( pub fn evaluate_alerts(
rules: &[AlertRule], rules: &[AlertRule],
states: &mut HashMap<String, AlertRuntimeState>, states: &mut HashMap<String, AlertRuntimeState>,
snapshot: &engine::EngineSnapshot,
sequence: u64, sequence: u64,
timestamp: Timestamp, timestamp: Timestamp,
text: &str, text: &str,
@@ -215,7 +222,9 @@ pub fn evaluate_alerts(
continue; continue;
} }
let state = states.entry(rule.id.clone()).or_default(); let state = states.entry(rule.id.clone()).or_default();
if let Some(message) = evaluate_condition(&rule.condition, state, sequence, timestamp, text, &rule.name) { if let Some(message) =
evaluate_condition(&rule.condition, state, snapshot, sequence, timestamp, text, &rule.name)
{
fired.push((rule.clone(), message)); fired.push((rule.clone(), message));
} }
} }
@@ -225,6 +234,7 @@ pub fn evaluate_alerts(
fn evaluate_condition( fn evaluate_condition(
condition: &AlertCondition, condition: &AlertCondition,
state: &mut AlertRuntimeState, state: &mut AlertRuntimeState,
snapshot: &engine::EngineSnapshot,
sequence: u64, sequence: u64,
timestamp: Timestamp, timestamp: Timestamp,
text: &str, text: &str,
@@ -240,13 +250,15 @@ fn evaluate_condition(
} }
}, },
AlertCondition::Threshold { query, op, threshold, window_ms } => { AlertCondition::Threshold { query, op, threshold, window_ms } => {
if !state.threshold_backfilled {
state.threshold_window = backfill_window(snapshot, query, sequence, timestamp, *window_ms);
state.threshold_backfilled = true;
}
match eval_alert_query(query, sequence, timestamp, text) { match eval_alert_query(query, sequence, timestamp, text) {
Ok(true) => { Ok(true) => {
let now_ms = timestamp.as_millis(); state.threshold_window.push(timestamp, *window_ms);
state.threshold_matches.push_back(now_ms); let count = state.threshold_window.count() as f64;
prune_window(&mut state.threshold_matches, now_ms, *window_ms); op.matches(count.total_cmp(threshold)).then(|| {
let count = state.threshold_matches.len() as f64;
apply_compare_op(count.total_cmp(threshold), *op).then(|| {
format!("{rule_name}: match count {count} {op:?} threshold {threshold} (window {window_ms}ms)") format!("{rule_name}: match count {count} {op:?} threshold {threshold} (window {window_ms}ms)")
}) })
} }
@@ -285,23 +297,27 @@ fn evaluate_condition(
if !matched_a && !matched_b { if !matched_a && !matched_b {
return None; return None;
} }
let now_ms = timestamp.as_millis(); if !state.ratio_a_backfilled {
state.ratio_a_window = backfill_window(snapshot, query_a, sequence, timestamp, *window_ms);
state.ratio_a_backfilled = true;
}
if !state.ratio_b_backfilled {
state.ratio_b_window = backfill_window(snapshot, query_b, sequence, timestamp, *window_ms);
state.ratio_b_backfilled = true;
}
if matched_a { if matched_a {
state.ratio_a_matches.push_back(now_ms); state.ratio_a_window.push(timestamp, *window_ms);
prune_window(&mut state.ratio_a_matches, now_ms, *window_ms);
} }
if matched_b { if matched_b {
state.ratio_b_matches.push_back(now_ms); state.ratio_b_window.push(timestamp, *window_ms);
prune_window(&mut state.ratio_b_matches, now_ms, *window_ms);
} }
let count_a = state.ratio_a_matches.len() as f64; let count_a = state.ratio_a_window.count() as f64;
let count_b = state.ratio_b_matches.len() as f64; let count_b = state.ratio_b_window.count() as f64;
if count_b == 0.0 { if count_b == 0.0 {
// Zero denominator: skip rather than treat the ratio as infinite.
return None; return None;
} }
let observed = count_a / count_b; let observed = count_a / count_b;
apply_compare_op(observed.total_cmp(ratio), *op).then(|| { op.matches(observed.total_cmp(ratio)).then(|| {
format!( format!(
"{rule_name}: ratio {observed:.3} ({count_a}/{count_b}) {op:?} {ratio} (window {window_ms}ms)" "{rule_name}: ratio {observed:.3} ({count_a}/{count_b}) {op:?} {ratio} (window {window_ms}ms)"
) )
@@ -310,9 +326,6 @@ fn evaluate_condition(
} }
} }
/// Delivers to the single, process-wide ntfy sink configured in Settings. Errors are only logged
/// (no retry queue); spawned off the ingest hot path so a slow/unreachable sink never blocks ingestion.
/// A silently no-op'd delivery (no topic configured) is expected — a rule can be defined before the sink is.
pub async fn deliver_for_rule(rule_name: String, message: String, topic: Option<String>, server: Option<String>) { pub async fn deliver_for_rule(rule_name: String, message: String, topic: Option<String>, server: Option<String>) {
let Some(topic) = topic else { let Some(topic) = topic else {
return; return;
@@ -323,7 +336,6 @@ pub async fn deliver_for_rule(rule_name: String, message: String, topic: Option<
} }
} }
/// Standalone and `AlertRule`-free so a "send test notification" endpoint can call it directly.
pub async fn deliver_ntfy(server: &str, topic: &str, message: &str) -> Result<(), String> { pub async fn deliver_ntfy(server: &str, topic: &str, message: &str) -> Result<(), String> {
let url = format!("{}/{}", server.trim_end_matches('/'), topic); let url = format!("{}/{}", server.trim_end_matches('/'), topic);
let response = reqwest::Client::new() let response = reqwest::Client::new()
@@ -417,12 +429,13 @@ mod tests {
fn immediate_fires_on_matching_line_only() { fn immediate_fires_on_matching_line_only() {
let rules = vec![immediate_rule(r#"regex("error")"#)]; let rules = vec![immediate_rule(r#"regex("error")"#)];
let mut states = HashMap::new(); let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
let (seq, ts, text) = line("all good", 0, 0); let (seq, ts, text) = line("all good", 0, 0);
assert!(evaluate_alerts(&rules, &mut states, seq, ts, text).is_empty()); assert!(evaluate_alerts(&rules, &mut states, &index, seq, ts, text).is_empty());
let (seq, ts, text) = line("an error happened", 1, 1); let (seq, ts, text) = line("an error happened", 1, 1);
let fired = evaluate_alerts(&rules, &mut states, seq, ts, text); let fired = evaluate_alerts(&rules, &mut states, &index, seq, ts, text);
assert_eq!(fired.len(), 1); assert_eq!(fired.len(), 1);
assert!(fired[0].1.contains("an error happened")); assert!(fired[0].1.contains("an error happened"));
} }
@@ -432,8 +445,9 @@ mod tests {
let mut rule = immediate_rule(r#"regex("error")"#); let mut rule = immediate_rule(r#"regex("error")"#);
rule.enabled = false; rule.enabled = false;
let mut states = HashMap::new(); let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
let (seq, ts, text) = line("error!", 0, 0); let (seq, ts, text) = line("error!", 0, 0);
assert!(evaluate_alerts(&[rule], &mut states, seq, ts, text).is_empty()); assert!(evaluate_alerts(&[rule], &mut states, &index, seq, ts, text).is_empty());
} }
#[test] #[test]
@@ -450,14 +464,14 @@ mod tests {
}, },
}; };
let mut states = HashMap::new(); let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
for (seq, ms) in [(0u64, 0i64), (1, 1000)] { for (seq, ms) in [(0u64, 0i64), (1, 1000)] {
let fired = evaluate_alerts(&[rule.clone()], &mut states, seq, Timestamp::from_millis(ms), "error"); let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, seq, Timestamp::from_millis(ms), "error");
assert!(fired.is_empty(), "should not fire before threshold is crossed"); assert!(fired.is_empty(), "should not fire before threshold is crossed");
} }
let fired = evaluate_alerts(&[rule.clone()], &mut states, 2, Timestamp::from_millis(2000), "error"); let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 2, Timestamp::from_millis(2000), "error");
assert_eq!(fired.len(), 1); assert_eq!(fired.len(), 1);
// Stays quiet for a non-matching line even though the window is full. let fired = evaluate_alerts(&[rule], &mut states, &index, 3, Timestamp::from_millis(2100), "fine");
let fired = evaluate_alerts(&[rule], &mut states, 3, Timestamp::from_millis(2100), "fine");
assert!(fired.is_empty()); assert!(fired.is_empty());
} }
@@ -475,9 +489,9 @@ mod tests {
}, },
}; };
let mut states = HashMap::new(); let mut states = HashMap::new();
assert!(evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "error").is_empty()); let index = engine::EngineSnapshot::default();
// Second match arrives well after the first has aged out of the window. assert!(evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "error").is_empty());
assert!(evaluate_alerts(&[rule], &mut states, 1, Timestamp::from_millis(5_000), "error").is_empty()); assert!(evaluate_alerts(&[rule], &mut states, &index, 1, Timestamp::from_millis(5_000), "error").is_empty());
} }
#[test] #[test]
@@ -492,11 +506,12 @@ mod tests {
}, },
}; };
let mut states = HashMap::new(); let mut states = HashMap::new();
let fired = evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "agent=curl"); let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "agent=curl");
assert_eq!(fired.len(), 1); assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule.clone()], &mut states, 1, Timestamp::from_millis(1), "agent=curl"); let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 1, Timestamp::from_millis(1), "agent=curl");
assert!(fired.is_empty()); assert!(fired.is_empty());
let fired = evaluate_alerts(&[rule], &mut states, 2, Timestamp::from_millis(2), "agent=wget"); let fired = evaluate_alerts(&[rule], &mut states, &index, 2, Timestamp::from_millis(2), "agent=wget");
assert_eq!(fired.len(), 1); assert_eq!(fired.len(), 1);
} }
@@ -515,16 +530,33 @@ mod tests {
}, },
}; };
let mut states = HashMap::new(); let mut states = HashMap::new();
// Only an "error" so far; no "request" yet, so denominator is 0 -> skip. let index = engine::EngineSnapshot::default();
assert!(evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "error").is_empty()); assert!(evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "error").is_empty());
// One request: ratio 1/1 = 1.0 >= 0.5 -> fires. let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 1, Timestamp::from_millis(1), "request");
let fired = evaluate_alerts(&[rule.clone()], &mut states, 1, Timestamp::from_millis(1), "request");
assert_eq!(fired.len(), 1); assert_eq!(fired.len(), 1);
// A second request: ratio 1/2 = 0.5 still >= 0.5 -> fires again. let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 2, Timestamp::from_millis(2), "request");
let fired = evaluate_alerts(&[rule.clone()], &mut states, 2, Timestamp::from_millis(2), "request");
assert_eq!(fired.len(), 1); assert_eq!(fired.len(), 1);
// A third request with no more errors: ratio drops to 1/3 -> no longer fires. let fired = evaluate_alerts(&[rule], &mut states, &index, 3, Timestamp::from_millis(3), "request");
let fired = evaluate_alerts(&[rule], &mut states, 3, Timestamp::from_millis(3), "request"); assert!(fired.is_empty());
}
#[test]
fn ottyl_fmt_alert_query_matches_detected_format() {
let rule = immediate_rule("ottyl_fmt==json");
let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(
&[rule.clone()],
&mut states,
&index,
0,
Timestamp::from_millis(0),
r#"{"level":"error"}"#,
);
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule], &mut states, &index, 1, Timestamp::from_millis(1), "plain text");
assert!(fired.is_empty()); assert!(fired.is_empty());
} }
@@ -532,8 +564,8 @@ mod tests {
fn metric_comparison_in_alert_query_is_rejected_not_panicking() { fn metric_comparison_in_alert_query_is_rejected_not_panicking() {
let rule = immediate_rule("some_metric[param]==5"); let rule = immediate_rule("some_metric[param]==5");
let mut states = HashMap::new(); let mut states = HashMap::new();
// Should log a warning and simply not fire, never panic. let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(&[rule], &mut states, 0, Timestamp::from_millis(0), "anything"); let fired = evaluate_alerts(&[rule], &mut states, &index, 0, Timestamp::from_millis(0), "anything");
assert!(fired.is_empty()); assert!(fired.is_empty());
} }
src/engine/data_value.rs
+96
View File
@@ -0,0 +1,96 @@
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::timestamp::Timestamp;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ParamDeclaration {
pub name: String,
pub message_match_regex: String,
pub capture_group: String,
#[serde(default)]
pub json_field_path: Option<String>,
pub kind: MetricPrimitiveKind,
#[serde(default)]
pub elapsed_since_last: bool,
#[serde(default)]
pub auto_derived: bool,
#[serde(default)]
pub log_stream_gated: bool,
}
pub fn extract_json_field(text: &str, field_path: &str) -> Option<String> {
let root: serde_json::Value = serde_json::from_str(text).ok()?;
let mut current = root.as_object()?;
let mut segments = field_path.split('.').peekable();
loop {
let segment = segments.next()?;
let next_value = current.get(segment)?;
if segments.peek().is_none() {
return match next_value {
serde_json::Value::String(s) => Some(s.clone()),
serde_json::Value::Number(n) => Some(n.to_string()),
serde_json::Value::Bool(b) => Some(b.to_string()),
_ => None,
};
}
current = next_value.as_object()?;
}
}
pub fn flatten_json_object(text: &str) -> Vec<(String, String)> {
let Ok(serde_json::Value::Object(root)) = serde_json::from_str::<serde_json::Value>(text) else {
return Vec::new();
};
let mut results = Vec::new();
flatten_json_object_into(&root, "", &mut results);
results
}
fn flatten_json_object_into(
object: &serde_json::Map<String, serde_json::Value>,
prefix: &str,
results: &mut Vec<(String, String)>,
) {
for (key, value) in object {
let path = if prefix.is_empty() { key.clone() } else { format!("{prefix}.{key}") };
match value {
serde_json::Value::String(s) => results.push((path, s.clone())),
serde_json::Value::Number(n) => results.push((path, n.to_string())),
serde_json::Value::Bool(b) => results.push((path, b.to_string())),
serde_json::Value::Object(nested) => flatten_json_object_into(nested, &path, results),
serde_json::Value::Null | serde_json::Value::Array(_) => {}
}
}
}
#[allow(dead_code)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MetricPrimitiveKind {
String,
Number,
Timestamp,
Duration,
Boolean,
}
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize)]
pub enum DataValue {
LogLiteral { text: String },
ParamDeclaration(ParamDeclaration),
MetricDeclaration {
message_match_regex: String,
label_params: Vec<String>,
value_param: Option<String>,
matched_labels: Option<BTreeMap<String, String>>,
unmatched_labels: Option<BTreeMap<String, String>>,
},
BucketValue {
values: Vec<(u64, Timestamp, MetricPrimitive)>,
},
LastEventTimestamp {
timestamp: Timestamp,
},
}
src/filter_datetime.rs → src/engine/filter_datetime.rs
+1 -19
View File
@@ -1,21 +1,12 @@
//! Datetime parsing for the filter query language's timestamp comparisons: relative
//! humantime, time-of-day (resolved against the UTC calendar day of `now`), or a
//! full date/datetime literal.
// `#![allow(dead_code)]` stays since most of this module's surface is still only
// exercised by its own unit tests below.
#![allow(dead_code)] #![allow(dead_code)]
use chrono::{DateTime, NaiveDate, NaiveTime, TimeZone, Utc}; use chrono::{DateTime, NaiveDate, NaiveTime, TimeZone, Utc};
use crate::timestamp::Timestamp; use crate::engine::timestamp::Timestamp;
/// Error returned when a datetime filter literal can't be resolved to a [`Timestamp`].
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
pub enum DateTimeParseError { pub enum DateTimeParseError {
/// The input matched a supported format's shape but its value was invalid, e.g. `25:99`.
Invalid { input: String, reason: String }, Invalid { input: String, reason: String },
/// The input didn't match any supported format at all.
Unparseable(String), Unparseable(String),
} }
@@ -37,8 +28,6 @@ impl std::fmt::Display for DateTimeParseError {
impl std::error::Error for DateTimeParseError {} impl std::error::Error for DateTimeParseError {}
/// Parses a filter datetime literal into an absolute [`Timestamp`]; `now` is a parameter
/// (rather than calling `Timestamp::now()` internally) so callers can pass a fixed value in tests.
pub fn parse_datetime(input: &str, now: Timestamp) -> Result<Timestamp, DateTimeParseError> { pub fn parse_datetime(input: &str, now: Timestamp) -> Result<Timestamp, DateTimeParseError> {
let trimmed = input.trim(); let trimmed = input.trim();
if trimmed.is_empty() { if trimmed.is_empty() {
@@ -60,8 +49,6 @@ pub fn parse_datetime(input: &str, now: Timestamp) -> Result<Timestamp, DateTime
Err(DateTimeParseError::Unparseable(input.to_string())) Err(DateTimeParseError::Unparseable(input.to_string()))
} }
/// Tries to interpret `input` as a signed relative humantime duration (`5h`, `-3m`);
/// returns `None` if it doesn't look like one at all, so the caller can try other formats.
fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTimeParseError>> { fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTimeParseError>> {
let (sign, rest) = match input.strip_prefix('-') { let (sign, rest) = match input.strip_prefix('-') {
Some(rest) => (-1i128, rest), Some(rest) => (-1i128, rest),
@@ -71,7 +58,6 @@ fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTim
}, },
}; };
// Require a digit followed by an alphabetic unit so date-like inputs (e.g. "2024-01-15") fall through to the date parser instead of erroring here as an invalid duration.
let looks_like_duration = rest let looks_like_duration = rest
.chars() .chars()
.next() .next()
@@ -97,9 +83,7 @@ fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTim
} }
} }
/// Tries to interpret `input` as a full date or RFC3339-ish datetime literal.
fn try_full_datetime(input: &str) -> Option<Result<Timestamp, DateTimeParseError>> { fn try_full_datetime(input: &str) -> Option<Result<Timestamp, DateTimeParseError>> {
// Only attempt this path for inputs shaped like a date, e.g. "2024-01-15".
let looks_like_date = input.len() >= 8 && input.as_bytes()[4] == b'-'; let looks_like_date = input.len() >= 8 && input.as_bytes()[4] == b'-';
if !looks_like_date { if !looks_like_date {
return None; return None;
@@ -123,7 +107,6 @@ fn try_full_datetime(input: &str) -> Option<Result<Timestamp, DateTimeParseError
} }
} }
/// Tries to interpret `input` as a bare time-of-day, resolved against the UTC calendar day of `now`.
fn try_time_of_day( fn try_time_of_day(
input: &str, input: &str,
now: Timestamp, now: Timestamp,
@@ -147,7 +130,6 @@ fn try_time_of_day(
let now_millis = now.as_millis(); let now_millis = now.as_millis();
let now_dt = Utc.timestamp_millis_opt(now_millis).single().unwrap_or_else(|| { let now_dt = Utc.timestamp_millis_opt(now_millis).single().unwrap_or_else(|| {
// Fall back to the epoch date; shouldn't happen since `Timestamp` is always a valid instant.
Utc.timestamp_millis_opt(0).single().expect("epoch is valid") Utc.timestamp_millis_opt(0).single().expect("epoch is valid")
}); });
let today = now_dt.date_naive(); let today = now_dt.date_naive();
src/engine/filter_parser.pest
+30
View File
@@ -0,0 +1,30 @@
WHITESPACE = _{ " " | "\t" | "\n" | "\r" }
file = { SOI ~ or_expr ~ EOI }
or_expr = { and_expr ~ (or_kw ~ and_expr)* }
and_expr = { term ~ (and_kw ~ term)* }
term = _{ regex_expr | comparison | "(" ~ or_expr ~ ")" }
and_kw = @{ "and" ~ !(ASCII_ALPHANUMERIC | "_") }
or_kw = @{ "or" ~ !(ASCII_ALPHANUMERIC | "_") }
regex_expr = { "regex" ~ "(" ~ string_literal ~ ")" }
string_literal = ${ "\"" ~ inner_str ~ "\"" }
inner_str = @{ (escaped_char | !("\"" | "\\") ~ ANY)* }
escaped_char = @{ "\\" ~ ANY }
comparison = _{ json_comparison | metric_comparison }
json_comparison = { "json" ~ "[" ~ param ~ "]" ~ op ~ value }
metric_comparison = { metric ~ ("[" ~ param ~ "]")? ~ op ~ value }
metric = @{ ident }
param = @{ ident }
ident = @{ (ASCII_ALPHANUMERIC | "_")+ }
op = { "<=" | ">=" | "==" | "!=" | "<" | ">" }
value = @{ (!(WHITESPACE | ")") ~ ANY)+ }
src/filter_parser.rs → src/engine/filter_parser.rs
+29 -32
View File
@@ -1,5 +1,3 @@
//! FilterBar query language: grammar (see `src/filter_parser.pest`), AST and parser.
//! Evaluating the AST against real log data is left to `engine.rs`.
#![allow(dead_code)] #![allow(dead_code)]
use pest::Parser; use pest::Parser;
@@ -8,23 +6,18 @@ use pest_derive::Parser as PestParser;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
#[derive(PestParser)] #[derive(PestParser)]
#[grammar = "filter_parser.pest"] #[grammar = "engine/filter_parser.pest"]
struct FilterGrammar; struct FilterGrammar;
/// The parsed AST for a FilterBar query.
#[derive(Debug, Clone, PartialEq)] #[derive(Debug, Clone, PartialEq)]
pub enum FilterExpr { pub enum FilterExpr {
/// `regex("...")` — raw-text regex search over log lines.
Regex(String), Regex(String),
/// `metric[param] <op> value` — a single metric comparison.
Comparison { Comparison {
metric: String, metric: String,
param: Option<String>, param: Option<String>,
op: CompareOp, op: CompareOp,
value: FilterValue, value: FilterValue,
}, },
/// `json[field] <op> value` — a comparison against a field of the line's parsed JSON.
/// Lines that aren't valid JSON (or lack the field) simply don't match.
JsonComparison { JsonComparison {
field: String, field: String,
op: CompareOp, op: CompareOp,
@@ -34,8 +27,6 @@ pub enum FilterExpr {
Or(Box<FilterExpr>, Box<FilterExpr>), Or(Box<FilterExpr>, Box<FilterExpr>),
} }
/// Comparison operators for metric comparisons; also reused by `crate::alerts::AlertCondition`
/// for threshold/ratio comparisons (hence `Serialize`/`Deserialize`).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")] #[serde(rename_all = "lowercase")]
pub enum CompareOp { pub enum CompareOp {
@@ -47,15 +38,12 @@ pub enum CompareOp {
Ne, Ne,
} }
/// The right-hand side of a comparison: values that parse as a number become
/// [`FilterValue::Number`], everything else is kept as an opaque [`FilterValue::Raw`] string.
#[derive(Debug, Clone, PartialEq)] #[derive(Debug, Clone, PartialEq)]
pub enum FilterValue { pub enum FilterValue {
Number(f64), Number(f64),
Raw(String), Raw(String),
} }
/// A located, human-readable parse error.
#[derive(Debug, Clone, PartialEq)] #[derive(Debug, Clone, PartialEq)]
pub struct FilterParseError { pub struct FilterParseError {
message: String, message: String,
@@ -79,12 +67,10 @@ impl std::error::Error for FilterParseError {}
impl From<pest::error::Error<Rule>> for FilterParseError { impl From<pest::error::Error<Rule>> for FilterParseError {
fn from(err: pest::error::Error<Rule>) -> Self { fn from(err: pest::error::Error<Rule>) -> Self {
// pest's Display already renders a located message, so use it and add a short prefix.
FilterParseError::new(format!("failed to parse filter expression: {err}")) FilterParseError::new(format!("failed to parse filter expression: {err}"))
} }
} }
/// Parse a FilterBar query string into a [`FilterExpr`] AST.
pub fn parse(input: &str) -> Result<FilterExpr, FilterParseError> { pub fn parse(input: &str) -> Result<FilterExpr, FilterParseError> {
let mut pairs = FilterGrammar::parse(Rule::file, input)?; let mut pairs = FilterGrammar::parse(Rule::file, input)?;
@@ -102,7 +88,6 @@ pub fn parse(input: &str) -> Result<FilterExpr, FilterParseError> {
fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> { fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
debug_assert_eq!(pair.as_rule(), Rule::or_expr); debug_assert_eq!(pair.as_rule(), Rule::or_expr);
// `or_kw` is atomic so it's not silent, showing up as a sibling pair between `and_expr` operands — skip it rather than assuming a fixed alternating structure.
let mut and_exprs = pair let mut and_exprs = pair
.into_inner() .into_inner()
.filter(|p| p.as_rule() != Rule::or_kw); .filter(|p| p.as_rule() != Rule::or_kw);
@@ -119,7 +104,6 @@ fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
fn build_and_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> { fn build_and_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
debug_assert_eq!(pair.as_rule(), Rule::and_expr); debug_assert_eq!(pair.as_rule(), Rule::and_expr);
// Same reasoning as `build_or_expr` for `and_kw`.
let mut terms = pair let mut terms = pair
.into_inner() .into_inner()
.filter(|p| p.as_rule() != Rule::and_kw); .filter(|p| p.as_rule() != Rule::and_kw);
@@ -237,7 +221,6 @@ fn parse_value(s: &str) -> FilterValue {
} }
} }
/// Unescape a regex string literal's inner contents (`\"` -> `"`, `\\` -> `\`, `\x` -> `x`).
fn unescape(s: &str) -> String { fn unescape(s: &str) -> String {
let mut out = String::with_capacity(s.len()); let mut out = String::with_capacity(s.len());
let mut chars = s.chars(); let mut chars = s.chars();
@@ -284,12 +267,12 @@ mod tests {
} }
#[test] #[test]
fn parses_reserved_metric_sequence() { fn parses_reserved_metric_ottyl_seq() {
let expr = parse("sequence<5").expect("should parse"); let expr = parse("ottyl_seq<5").expect("should parse");
assert_eq!( assert_eq!(
expr, expr,
FilterExpr::Comparison { FilterExpr::Comparison {
metric: "sequence".to_string(), metric: "ottyl_seq".to_string(),
param: None, param: None,
op: CompareOp::Lt, op: CompareOp::Lt,
value: FilterValue::Number(5.0), value: FilterValue::Number(5.0),
@@ -298,12 +281,12 @@ mod tests {
} }
#[test] #[test]
fn parses_reserved_metric_timestamp_as_raw_value() { fn parses_reserved_metric_ottyl_ts_as_raw_value() {
let expr = parse("timestamp>12:00").expect("should parse"); let expr = parse("ottyl_ts>12:00").expect("should parse");
assert_eq!( assert_eq!(
expr, expr,
FilterExpr::Comparison { FilterExpr::Comparison {
metric: "timestamp".to_string(), metric: "ottyl_ts".to_string(),
param: None, param: None,
op: CompareOp::Gt, op: CompareOp::Gt,
value: FilterValue::Raw("12:00".to_string()), value: FilterValue::Raw("12:00".to_string()),
@@ -311,6 +294,20 @@ mod tests {
); );
} }
#[test]
fn parses_reserved_metric_ottyl_fmt() {
let expr = parse("ottyl_fmt==json").expect("should parse");
assert_eq!(
expr,
FilterExpr::Comparison {
metric: "ottyl_fmt".to_string(),
param: None,
op: CompareOp::Eq,
value: FilterValue::Raw("json".to_string()),
}
);
}
#[test] #[test]
fn parses_json_field_comparison() { fn parses_json_field_comparison() {
let expr = parse("json[level]==error").expect("should parse"); let expr = parse("json[level]==error").expect("should parse");
@@ -339,7 +336,7 @@ mod tests {
#[test] #[test]
fn json_field_composes_with_and_or() { fn json_field_composes_with_and_or() {
let expr = parse(r#"json[level]==error and sequence<5"#).expect("should parse"); let expr = parse(r#"json[level]==error and ottyl_seq<5"#).expect("should parse");
let expected = FilterExpr::And( let expected = FilterExpr::And(
Box::new(FilterExpr::JsonComparison { Box::new(FilterExpr::JsonComparison {
field: "level".to_string(), field: "level".to_string(),
@@ -347,7 +344,7 @@ mod tests {
value: FilterValue::Raw("error".to_string()), value: FilterValue::Raw("error".to_string()),
}), }),
Box::new(FilterExpr::Comparison { Box::new(FilterExpr::Comparison {
metric: "sequence".to_string(), metric: "ottyl_seq".to_string(),
param: None, param: None,
op: CompareOp::Lt, op: CompareOp::Lt,
value: FilterValue::Number(5.0), value: FilterValue::Number(5.0),
@@ -358,21 +355,21 @@ mod tests {
#[test] #[test]
fn parses_and_or_with_and_binding_tighter() { fn parses_and_or_with_and_binding_tighter() {
let expr = parse(r#"regex("error") and sequence<5 or timestamp>12:00"#) let expr = parse(r#"regex("error") and ottyl_seq<5 or ottyl_ts>12:00"#)
.expect("should parse"); .expect("should parse");
let expected = FilterExpr::Or( let expected = FilterExpr::Or(
Box::new(FilterExpr::And( Box::new(FilterExpr::And(
Box::new(FilterExpr::Regex("error".to_string())), Box::new(FilterExpr::Regex("error".to_string())),
Box::new(FilterExpr::Comparison { Box::new(FilterExpr::Comparison {
metric: "sequence".to_string(), metric: "ottyl_seq".to_string(),
param: None, param: None,
op: CompareOp::Lt, op: CompareOp::Lt,
value: FilterValue::Number(5.0), value: FilterValue::Number(5.0),
}), }),
)), )),
Box::new(FilterExpr::Comparison { Box::new(FilterExpr::Comparison {
metric: "timestamp".to_string(), metric: "ottyl_ts".to_string(),
param: None, param: None,
op: CompareOp::Gt, op: CompareOp::Gt,
value: FilterValue::Raw("12:00".to_string()), value: FilterValue::Raw("12:00".to_string()),
@@ -384,20 +381,20 @@ mod tests {
#[test] #[test]
fn parenthesized_expression_overrides_precedence() { fn parenthesized_expression_overrides_precedence() {
let expr = parse(r#"regex("error") and (sequence<5 or timestamp>12:00)"#) let expr = parse(r#"regex("error") and (ottyl_seq<5 or ottyl_ts>12:00)"#)
.expect("should parse"); .expect("should parse");
let expected = FilterExpr::And( let expected = FilterExpr::And(
Box::new(FilterExpr::Regex("error".to_string())), Box::new(FilterExpr::Regex("error".to_string())),
Box::new(FilterExpr::Or( Box::new(FilterExpr::Or(
Box::new(FilterExpr::Comparison { Box::new(FilterExpr::Comparison {
metric: "sequence".to_string(), metric: "ottyl_seq".to_string(),
param: None, param: None,
op: CompareOp::Lt, op: CompareOp::Lt,
value: FilterValue::Number(5.0), value: FilterValue::Number(5.0),
}), }),
Box::new(FilterExpr::Comparison { Box::new(FilterExpr::Comparison {
metric: "timestamp".to_string(), metric: "ottyl_ts".to_string(),
param: None, param: None,
op: CompareOp::Gt, op: CompareOp::Gt,
value: FilterValue::Raw("12:00".to_string()), value: FilterValue::Raw("12:00".to_string()),
src/engine/handle.rs
+302
View File
@@ -0,0 +1,302 @@
//! The left-right write/read plumbing around `EngineSnapshot`, the actor loop that owns it,
//! and the `EngineHandle` other tasks use to talk to that actor.
use crate::consts::{
INGEST_LAG_WATERMARK_MAX_ENTRIES, INGEST_LAG_WATERMARK_WINDOW_MS,
};
use crate::engine::alerts::{self, AlertRuntimeState};
use crate::engine::data_value::DataValue;
use crate::engine::index_key::IndexKey;
use crate::engine::message::EngineMessage;
use crate::engine::metric_primitive::{MetricPrimitive, Number};
use crate::engine::snapshot::params::log_stream_header_transition;
use crate::engine::snapshot::{approx_log_bytes, EngineSnapshot};
use crate::engine::timestamp::Timestamp;
use crate::engine::{ClearParamsResult, IngestedLine};
use crate::config::SettingsHandle;
use left_right::Absorb;
use std::collections::{BTreeMap, HashMap};
use tokio::sync::{mpsc, oneshot};
struct EngineOp(Box<dyn FnMut(&mut EngineSnapshot) + Send + 'static>);
impl Absorb<EngineOp> for EngineSnapshot {
fn absorb_first(&mut self, operation: &mut EngineOp, _other: &Self) {
(operation.0)(self);
}
fn drop_first(self: Box<Self>) {}
fn sync_with(&mut self, first: &Self) {
*self = first.clone();
}
}
struct IndexWriter {
handle: left_right::WriteHandle<EngineSnapshot, EngineOp>,
}
impl IndexWriter {
fn mutate(&mut self, op: impl FnMut(&mut EngineSnapshot) + Send + 'static) {
self.handle.append(EngineOp(Box::new(op)));
self.handle.publish();
}
fn peek(&mut self) -> Option<left_right::ReadGuard<'_, EngineSnapshot>> {
self.handle.enter()
}
}
pub struct IndexReadHandle(left_right::ReadHandle<EngineSnapshot>);
impl IndexReadHandle {
pub(crate) fn enter(&self) -> Option<left_right::ReadGuard<'_, EngineSnapshot>> {
self.0.enter()
}
}
pub struct IndexReader {
factory: left_right::ReadHandleFactory<EngineSnapshot>,
}
impl Clone for IndexReader {
fn clone(&self) -> Self {
IndexReader { factory: self.factory.clone() }
}
}
impl IndexReader {
pub(crate) fn handle(&self) -> IndexReadHandle {
IndexReadHandle(self.factory.handle())
}
}
pub struct EngineHandle {
pub tx: mpsc::Sender<EngineMessage>,
pub reader: IndexReader,
}
impl Clone for EngineHandle {
fn clone(&self) -> Self {
EngineHandle { tx: self.tx.clone(), reader: self.reader.clone() }
}
}
impl EngineHandle {
pub async fn barrier(&self) {
let (respond_to, response) = oneshot::channel();
if self.tx.send(EngineMessage::Barrier(respond_to)).await.is_ok() {
let _ = response.await;
}
}
}
pub fn spawn(settings: SettingsHandle, channel_capacity: usize) -> EngineHandle {
let (tx, rx) = mpsc::channel(channel_capacity);
let (write, read) = left_right::new::<EngineSnapshot, EngineOp>();
let reader = IndexReader { factory: read.factory() };
tokio::spawn(run(rx, settings, IndexWriter { handle: write }));
EngineHandle { tx, reader }
}
#[cfg(test)]
pub(crate) fn test_handle(channel_capacity: usize) -> (EngineHandle, mpsc::Receiver<EngineMessage>) {
let (tx, rx) = mpsc::channel(channel_capacity);
let (_write, read) = left_right::new::<EngineSnapshot, EngineOp>();
(EngineHandle { tx, reader: IndexReader { factory: read.factory() } }, rx)
}
async fn run(mut rx: mpsc::Receiver<EngineMessage>, settings: SettingsHandle, mut write: IndexWriter) {
let mut max_logs: Option<usize> = None;
let mut metric_sample_sequence: u64 = 0;
let mut alert_states: HashMap<String, AlertRuntimeState> = HashMap::new();
let mut log_stream_header: Option<crate::engine::logfmt::LogStreamHeader> = None;
let mut ingest_lag_samples: Vec<(Timestamp, i64)> = Vec::new();
while let Some(message) = rx.recv().await {
match message {
EngineMessage::Ingest(line) => {
let IngestedLine { sequence, timestamp, text } = line;
let processed_at = Timestamp::now();
record_ingest_lag(
&mut ingest_lag_samples,
processed_at,
(processed_at.as_millis() - timestamp.as_millis()).max(0),
);
let ingest_lag_high_watermark_ms =
ingest_lag_high_watermark_ms(&ingest_lag_samples, processed_at);
let is_new_header = crate::engine::logfmt::LogStreamHeader::detect(&text).is_some();
log_stream_header = log_stream_header_transition(&log_stream_header, &text);
let row_header = if is_new_header { None } else { log_stream_header.clone() };
let (rules, ntfy_topic, ntfy_server) = {
let settings = settings.lock().unwrap();
(settings.alert_rules.clone(), settings.ntfy_topic.clone(), settings.ntfy_server.clone())
};
let text_for_op = text.clone();
write.mutate(move |snap: &mut EngineSnapshot| {
snap.approx_bytes_total += approx_log_bytes(&text_for_op);
snap.index.insert(
IndexKey::IngestedLog { sequence, timestamp },
DataValue::LogLiteral { text: text_for_op.clone() },
);
snap.log_count += 1;
if let Some(header) = row_header.as_ref() {
snap.derive_log_stream_params_from_row(header, &text_for_op);
}
snap.auto_declare_params(&text_for_op);
snap.apply_all_metrics(sequence, timestamp, &text_for_op);
snap.update_sort_helper_indexes_on_ingest(sequence, timestamp, &text_for_op);
snap.purge_stale_sort_helper_indexes();
snap.evict_excess_logs(max_logs);
snap.ingest_lag_high_watermark_ms = ingest_lag_high_watermark_ms;
});
let fired = write
.peek()
.map(|guard| {
alerts::evaluate_alerts(&rules, &mut alert_states, &guard, sequence, timestamp, &text)
})
.unwrap_or_default();
for (rule, message) in fired {
tokio::spawn(alerts::deliver_for_rule(rule.name, message, ntfy_topic.clone(), ntfy_server.clone()));
}
}
EngineMessage::IngestMetricSample(sample) => {
let labels: BTreeMap<String, MetricPrimitive> = sample
.labels
.into_iter()
.map(|(name, value)| (name, MetricPrimitive::String(value)))
.collect();
let metric_name = sample.metric_name;
let sequence = metric_sample_sequence;
let timestamp = sample.timestamp;
let value = MetricPrimitive::Number(Number::Float(sample.value));
write.mutate(move |snap: &mut EngineSnapshot| {
snap.push_bucket_value(&metric_name, labels.clone(), sequence, timestamp, value.clone());
});
metric_sample_sequence += 1;
}
EngineMessage::SetMaxLogs(new_max) => {
max_logs = new_max;
write.mutate(move |snap: &mut EngineSnapshot| {
snap.evict_excess_logs(new_max);
});
}
EngineMessage::DeclareParam { request, respond_to } => {
let result = write
.peek()
.map(|guard| guard.validate_param_declaration(&request))
.unwrap_or_else(|| Err("engine is shutting down".to_string()));
let succeeded = result.is_ok();
let name = request.name.clone();
let request_for_op = request.clone();
write.mutate(move |snap: &mut EngineSnapshot| {
snap.sort_helper_indexes.remove(&name);
if succeeded {
snap.insert_param_declaration(request_for_op.clone());
}
});
let _ = respond_to.send(result);
}
EngineMessage::RemoveParamDeclaration { name, respond_to } => {
let existed = write
.peek()
.is_some_and(|guard| guard.index.contains_key(&IndexKey::ParamDeclaration { name: name.clone() }));
let name_for_op = name.clone();
write.mutate(move |snap: &mut EngineSnapshot| {
snap.index.remove(&IndexKey::ParamDeclaration { name: name_for_op.clone() });
snap.sort_helper_indexes.remove(&name_for_op);
snap.param_regex_cache.remove(&name_for_op);
});
let _ = respond_to.send(existed);
}
EngineMessage::ClearParams { respond_to } => {
let mut scratch = EngineSnapshot::from_index(
write.peek().map(|guard| guard.index.clone()).unwrap_or_default(),
);
let removed = scratch.clear_all_param_declarations();
let (reextracted, logs_considered) = scratch.reextract_params_from_retained_logs();
write.mutate(move |snap: &mut EngineSnapshot| {
snap.clear_all_param_declarations();
snap.reextract_params_from_retained_logs();
snap.sort_helper_indexes.clear();
});
let _ = respond_to.send(ClearParamsResult { removed, reextracted, logs_considered });
}
EngineMessage::DeclareMetric { request, respond_to } => {
let result = write
.peek()
.map(|guard| guard.validate_metric_request(&request).map(|_| ()))
.unwrap_or_else(|| Err("engine is shutting down".to_string()));
let succeeded = result.is_ok();
let request_for_op = request.clone();
write.mutate(move |snap: &mut EngineSnapshot| {
if succeeded {
let _ = snap.declare_metric(request_for_op.clone());
}
});
let _ = respond_to.send(result);
}
EngineMessage::UpdateMetric { name, request, respond_to } => {
let result = write
.peek()
.map(|guard| guard.validate_update_metric(&name, &request).map(|_| ()))
.unwrap_or_else(|| Err("engine is shutting down".to_string()));
let succeeded = result.is_ok();
let existing_name = name.clone();
let request_for_op = request.clone();
write.mutate(move |snap: &mut EngineSnapshot| {
if succeeded {
let _ = snap.update_metric(&existing_name, request_for_op.clone());
}
});
let _ = respond_to.send(result);
}
EngineMessage::RemoveMetricDeclaration { name, respond_to } => {
let existed = write
.peek()
.is_some_and(|guard| guard.index.contains_key(&IndexKey::MetricDeclaration { name: name.clone() }));
let name_for_op = name.clone();
write.mutate(move |snap: &mut EngineSnapshot| {
snap.remove_metric_declaration(&name_for_op);
});
let _ = respond_to.send(existed);
}
EngineMessage::EnsureSortHelperIndex { param_name, respond_to } => {
write.mutate(move |snap: &mut EngineSnapshot| {
snap.ensure_sort_helper_index_built(&param_name);
});
let _ = respond_to.send(());
}
EngineMessage::Barrier(respond_to) => {
let _ = respond_to.send(());
}
}
}
}
pub(crate) fn record_ingest_lag(samples: &mut Vec<(Timestamp, i64)>, now: Timestamp, lag_ms: i64) {
samples.retain(|(recorded_at, _)| now.as_millis() - recorded_at.as_millis() < INGEST_LAG_WATERMARK_WINDOW_MS);
if samples.len() < INGEST_LAG_WATERMARK_MAX_ENTRIES {
samples.push((now, lag_ms));
return;
}
let Some((min_index, &(_, min_lag))) = samples.iter().enumerate().min_by_key(|(_, (_, lag))| *lag) else {
return;
};
if lag_ms > min_lag {
samples[min_index] = (now, lag_ms);
}
}
pub(crate) fn ingest_lag_high_watermark_ms(samples: &[(Timestamp, i64)], now: Timestamp) -> i64 {
samples
.iter()
.filter(|(recorded_at, _)| now.as_millis() - recorded_at.as_millis() < INGEST_LAG_WATERMARK_WINDOW_MS)
.map(|(_, lag)| *lag)
.max()
.unwrap_or(0)
}
src/engine/index_key.rs
+32
View File
@@ -0,0 +1,32 @@
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::timestamp::Timestamp;
use serde::Serialize;
use std::collections::BTreeMap;
#[allow(dead_code)]
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub enum IndexKey {
Start,
IngestedLog {
sequence: u64,
timestamp: Timestamp,
},
LogParamSplit,
ParamDeclaration {
name: String,
},
ParamMetricSplit,
MetricDeclaration {
name: String,
},
MetricBucketSplit,
Bucket {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
LastMetricEvent {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
End,
}
src/ingest.rs → src/engine/ingest.rs
+16 -38
View File
@@ -1,20 +1,13 @@
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::time::Duration;
use crate::engine::{EngineMessage, IngestedLine, MetricSample}; use crate::consts::{METRICS_POLL_INTERVAL, STDIN_POLL_INTERVAL as POLL_INTERVAL};
use crate::timestamp::Timestamp; use crate::engine::{EngineHandle, EngineMessage, IngestedLine, MetricSample};
use crate::engine::timestamp::Timestamp;
use tokio::fs::File; use tokio::fs::File;
use tokio::io::{AsyncBufReadExt, BufReader}; use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::sync::mpsc;
const POLL_INTERVAL: Duration = Duration::from_millis(200); pub async fn run(engine: EngineHandle) {
const METRICS_POLL_INTERVAL: Duration = Duration::from_secs(15);
/// Reads stdin line by line, sequences and timestamps each line, and forwards it
/// to the engine task.
pub async fn run(engine: mpsc::Sender<EngineMessage>) {
let stdin = tokio::io::stdin(); let stdin = tokio::io::stdin();
let mut lines = BufReader::new(stdin).lines(); let mut lines = BufReader::new(stdin).lines();
let mut sequence: u64 = 0; let mut sequence: u64 = 0;
@@ -27,13 +20,12 @@ pub async fn run(engine: mpsc::Sender<EngineMessage>) {
}; };
sequence += 1; sequence += 1;
if engine.send(EngineMessage::Ingest(line)).await.is_err() { if engine.tx.send(EngineMessage::Ingest(line)).await.is_err() {
break; break;
} }
} }
} }
/// Identifies a specific file instance on disk, so a recreated path can be told apart from the same file simply growing.
#[derive(PartialEq, Eq, Clone, Copy)] #[derive(PartialEq, Eq, Clone, Copy)]
struct FileId(#[cfg(unix)] u64, #[cfg(not(unix))] ()); struct FileId(#[cfg(unix)] u64, #[cfg(not(unix))] ());
@@ -56,10 +48,7 @@ async fn open(path: &Path) -> Option<(File, FileId)> {
Some((file, file_id(&meta))) Some((file, file_id(&meta)))
} }
/// Tails `path` like `tail -f`, resuming from the start if the file is deleted/recreated or pub async fn run_file(engine: EngineHandle, path: PathBuf) {
/// truncated in place rather than exiting or getting stuck; `sequence` keeps incrementing
/// across a recreation since it's ottyl's own line counter, not tied to the source file.
pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
let mut sequence: u64 = 0; let mut sequence: u64 = 0;
'reopen: loop { 'reopen: loop {
@@ -78,7 +67,6 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
let mut chunk = String::new(); let mut chunk = String::new();
match reader.read_line(&mut chunk).await { match reader.read_line(&mut chunk).await {
Ok(0) => { Ok(0) => {
// EOF: decide whether the file was truncated/recreated or just hasn't grown yet.
match tokio::fs::metadata(&path).await { match tokio::fs::metadata(&path).await {
Ok(meta) => { Ok(meta) => {
let new_id = file_id(&meta); let new_id = file_id(&meta);
@@ -87,9 +75,7 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
} }
id = new_id; id = new_id;
} }
Err(_) => { Err(_) => {}
// Path gone (mid-recreation); wait for it to reappear and reopen fresh.
}
} }
tokio::time::sleep(POLL_INTERVAL).await; tokio::time::sleep(POLL_INTERVAL).await;
} }
@@ -109,11 +95,10 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
}; };
sequence += 1; sequence += 1;
if engine.send(EngineMessage::Ingest(line)).await.is_err() { if engine.tx.send(EngineMessage::Ingest(line)).await.is_err() {
return; return;
} }
} }
// else: partial line at EOF (write in progress) — keep buffering.
} }
Err(_) => continue 'reopen, Err(_) => continue 'reopen,
} }
@@ -121,10 +106,7 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
} }
} }
/// Polls `url` on a fixed interval for a Prometheus/OpenMetrics-format response, pushing every parsed pub async fn run_metrics_poll(engine: EngineHandle, url: String) {
/// sample straight into the engine. Errors (unreachable endpoint, non-2xx response) are only logged —
/// polling keeps retrying on the next tick rather than exiting.
pub async fn run_metrics_poll(engine: mpsc::Sender<EngineMessage>, url: String) {
let client = reqwest::Client::new(); let client = reqwest::Client::new();
loop { loop {
match client.get(&url).send().await { match client.get(&url).send().await {
@@ -133,7 +115,7 @@ pub async fn run_metrics_poll(engine: mpsc::Sender<EngineMessage>, url: String)
Ok(body) => { Ok(body) => {
let now = Timestamp::now(); let now = Timestamp::now();
for sample in parse_openmetrics(&body, now) { for sample in parse_openmetrics(&body, now) {
if engine.send(EngineMessage::IngestMetricSample(sample)).await.is_err() { if engine.tx.send(EngineMessage::IngestMetricSample(sample)).await.is_err() {
return; return;
} }
} }
@@ -148,10 +130,6 @@ pub async fn run_metrics_poll(engine: mpsc::Sender<EngineMessage>, url: String)
} }
} }
/// Parses the Prometheus text exposition / OpenMetrics format: one sample per non-comment,
/// non-empty line, as `metric_name{label="value",...} value [timestamp_ms]` (labels and the
/// timestamp are both optional). A line that doesn't parse is skipped rather than aborting the
/// whole scrape.
fn parse_openmetrics(body: &str, now: Timestamp) -> Vec<MetricSample> { fn parse_openmetrics(body: &str, now: Timestamp) -> Vec<MetricSample> {
body.lines().filter_map(|line| parse_openmetrics_line(line, now)).collect() body.lines().filter_map(|line| parse_openmetrics_line(line, now)).collect()
} }
@@ -185,8 +163,6 @@ fn parse_openmetrics_line(line: &str, now: Timestamp) -> Option<MetricSample> {
Some(MetricSample { metric_name, labels, value, timestamp }) Some(MetricSample { metric_name, labels, value, timestamp })
} }
/// Splits on commas without handling an escaped comma inside a quoted label value — fine for the
/// common case this targets, not a full OpenMetrics-grammar parser.
fn parse_openmetrics_labels(raw: &str) -> BTreeMap<String, String> { fn parse_openmetrics_labels(raw: &str) -> BTreeMap<String, String> {
raw.split(',') raw.split(',')
.filter_map(|pair| pair.split_once('=')) .filter_map(|pair| pair.split_once('='))
@@ -238,7 +214,9 @@ mod openmetrics_tests {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use std::time::Duration;
use std::io::Write; use std::io::Write;
use crate::engine::test_handle;
use tokio::sync::mpsc; use tokio::sync::mpsc;
async fn recv_text(rx: &mut mpsc::Receiver<EngineMessage>) -> String { async fn recv_text(rx: &mut mpsc::Receiver<EngineMessage>) -> String {
@@ -258,7 +236,7 @@ mod tests {
let path = dir.path().join("app.log"); let path = dir.path().join("app.log");
std::fs::write(&path, "first\n").unwrap(); std::fs::write(&path, "first\n").unwrap();
let (tx, mut rx) = mpsc::channel(16); let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone())); tokio::spawn(run_file(tx, path.clone()));
assert_eq!(recv_text(&mut rx).await, "first"); assert_eq!(recv_text(&mut rx).await, "first");
@@ -274,7 +252,7 @@ mod tests {
let path = dir.path().join("app.log"); let path = dir.path().join("app.log");
std::fs::write(&path, "before-restart\n").unwrap(); std::fs::write(&path, "before-restart\n").unwrap();
let (tx, mut rx) = mpsc::channel(16); let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone())); tokio::spawn(run_file(tx, path.clone()));
assert_eq!(recv_text(&mut rx).await, "before-restart"); assert_eq!(recv_text(&mut rx).await, "before-restart");
@@ -291,7 +269,7 @@ mod tests {
let path = dir.path().join("app.log"); let path = dir.path().join("app.log");
std::fs::write(&path, "long line before truncation\n").unwrap(); std::fs::write(&path, "long line before truncation\n").unwrap();
let (tx, mut rx) = mpsc::channel(16); let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone())); tokio::spawn(run_file(tx, path.clone()));
assert_eq!(recv_text(&mut rx).await, "long line before truncation"); assert_eq!(recv_text(&mut rx).await, "long line before truncation");
@@ -312,7 +290,7 @@ mod tests {
let path = dir.path().join("app.log"); let path = dir.path().join("app.log");
std::fs::write(&path, "one\n").unwrap(); std::fs::write(&path, "one\n").unwrap();
let (tx, mut rx) = mpsc::channel(16); let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone())); tokio::spawn(run_file(tx, path.clone()));
let first = match tokio::time::timeout(Duration::from_secs(2), rx.recv()) let first = match tokio::time::timeout(Duration::from_secs(2), rx.recv())
src/engine/logfmt.rs
+524
View File
@@ -0,0 +1,524 @@
use nom::branch::alt;
use nom::bytes::complete::{escaped, tag, take_until, take_while1};
use nom::character::complete::{char, none_of, one_of};
use nom::error::{Error, ErrorKind};
use nom::sequence::delimited;
use nom::IResult;
use regex::Regex;
use std::sync::LazyLock;
use crate::consts::{COLUMN_SEPARATOR_PATTERN, COLUMN_VALUE_PATTERN};
fn key(input: &str) -> IResult<&str, &str> {
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '.' || c == '-')(input)
}
fn quoted_value(input: &str) -> IResult<&str, &str> {
delimited(
char('"'),
alt((escaped(none_of("\\\""), '\\', one_of("\"\\nt")), nom::combinator::success(""))),
char('"'),
)(input)
}
fn bareword_value(input: &str) -> IResult<&str, &str> {
take_while1(|c: char| !c.is_whitespace())(input)
}
fn looks_like_base64_padding_artifact(value: &str) -> bool {
value.starts_with('=')
}
fn pair(input: &str) -> IResult<&str, (&str, &str, bool)> {
let (input, k) = key(input)?;
let (input, _) = char('=')(input)?;
if let Ok((rest, v)) = quoted_value(input) {
Ok((rest, (k, v, true)))
} else {
let (rest, v) = bareword_value(input)?;
if looks_like_base64_padding_artifact(v) {
return Err(nom::Err::Error(Error::new(input, ErrorKind::Verify)));
}
Ok((rest, (k, v, false)))
}
}
pub fn extract_pairs(line: &str) -> Vec<(&str, &str, bool)> {
let mut results = Vec::new();
let mut rest = line;
while !rest.is_empty() {
match pair(rest) {
Ok((remaining, found)) => {
results.push(found);
rest = remaining;
}
Err(_) => {
let mut chars = rest.chars();
chars.next();
rest = chars.as_str();
}
}
}
results
}
static LOG_STREAM_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"^\d{4}-\d{2}-\d{2}\s+\S+\s+0x[0-9a-fA-F]+\s+\S+\s+0x[0-9a-fA-F]+\s+\d+\s+\d+\s+[^\s:]+:\s",
)
.expect("static log_stream regex must compile")
});
pub fn is_log_stream_line(line: &str) -> bool {
LOG_STREAM_RE.is_match(line)
}
fn bounded_column_value_pattern(width: usize) -> String {
if width <= 1 {
r"\S".to_string()
} else {
format!(r"\S(?:.{{0,{}}}\S)?", width - 2)
}
}
fn is_separator_run(run: &str) -> bool {
run.len() >= 2 || run.contains('\t')
}
static WHITESPACE_RUN: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"[ \t]+").expect("static regex must compile"));
fn split_columns_with_offsets(line: &str, max_columns: Option<usize>) -> Vec<(usize, &str)> {
let mut columns = Vec::new();
let mut start = 0usize;
let mut consumed = 0usize;
for m in WHITESPACE_RUN.find_iter(line) {
if max_columns.is_some_and(|max| consumed + 1 >= max) {
break;
}
let run = &line[m.start()..m.end()];
if is_separator_run(run) {
columns.push((start, &line[start..m.start()]));
start = m.end();
consumed += 1;
}
}
columns.push((start, &line[start..]));
columns
}
fn split_columns(line: &str, max_columns: Option<usize>) -> Vec<&str> {
split_columns_with_offsets(line, max_columns).into_iter().map(|(_, s)| s).collect()
}
static COLUMN_NAME: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[A-Za-z][A-Za-z0-9_]*$").expect("static regex must compile"));
#[derive(Clone)]
pub struct LogStreamColumn {
pub name: String,
start: usize,
}
#[derive(Clone)]
pub struct LogStreamHeader {
pub columns: Vec<LogStreamColumn>,
tab_delimited: bool,
}
fn char_slice(s: &str, start: usize, end: Option<usize>) -> &str {
if end.is_some_and(|end| end <= start) {
return "";
}
let mut indices = s.char_indices();
let byte_start = indices.nth(start).map(|(i, _)| i).unwrap_or(s.len());
let byte_end = match end {
Some(end) => s.char_indices().nth(end).map(|(i, _)| i).unwrap_or(s.len()),
None => s.len(),
};
&s[byte_start..byte_end.max(byte_start)]
}
impl LogStreamHeader {
pub fn detect(line: &str) -> Option<LogStreamHeader> {
if is_log_stream_line(line) {
return None;
}
let mut columns = Vec::new();
for (byte_start, slice) in split_columns_with_offsets(line, None) {
let trimmed = slice.trim();
if trimmed.is_empty() {
continue;
}
let leading_ws = slice.len() - slice.trim_start().len();
let start = line[..byte_start + leading_ws].chars().count();
columns.push(LogStreamColumn { name: trimmed.to_string(), start });
}
if columns.len() < 2 || !columns.iter().all(|c| COLUMN_NAME.is_match(&c.name)) {
return None;
}
let tab_delimited = WHITESPACE_RUN.find_iter(line).any(|m| m.as_str().contains('\t'));
Some(LogStreamHeader { columns, tab_delimited })
}
pub fn split_row<'a>(&self, line: &'a str) -> Vec<&'a str> {
if self.tab_delimited {
return split_columns(line, Some(self.columns.len() + 1));
}
let mut result = Vec::with_capacity(self.columns.len() + 1);
for (i, column) in self.columns.iter().enumerate() {
match self.columns.get(i + 1) {
Some(next) => result.push(char_slice(line, column.start, Some(next.start)).trim()),
None => {
let tail = char_slice(line, column.start, None);
let pieces = split_columns(tail, Some(2));
result.push(pieces[0].trim());
if let Some(message) = pieces.get(1) {
result.push(message);
}
}
}
}
result
}
pub fn column_pattern(&self, index: usize, capture_group: &str) -> String {
if self.tab_delimited {
let mut pattern = String::from("^");
for _ in 0..index {
pattern.push_str(COLUMN_VALUE_PATTERN);
pattern.push_str(COLUMN_SEPARATOR_PATTERN);
}
pattern.push_str(&format!("(?P<{capture_group}>{COLUMN_VALUE_PATTERN})"));
return pattern;
}
let start = self.columns[index].start;
let value_pattern = match self.columns.get(index + 1) {
Some(next) => bounded_column_value_pattern(next.start.saturating_sub(start)),
None => COLUMN_VALUE_PATTERN.to_string(),
};
format!("^.{{{start}}}(?P<{capture_group}>{value_pattern})")
}
pub fn message_pattern(&self, capture_group: &str) -> String {
if self.tab_delimited {
let mut pattern = String::from("^");
for _ in 0..self.columns.len() {
pattern.push_str(COLUMN_VALUE_PATTERN);
pattern.push_str(COLUMN_SEPARATOR_PATTERN);
}
pattern.push_str(&format!("(?P<{capture_group}>.*)$"));
return pattern;
}
let start = self.columns.last().map(|c| c.start).unwrap_or(0);
format!("^.{{{start}}}{COLUMN_VALUE_PATTERN}{COLUMN_SEPARATOR_PATTERN}(?P<{capture_group}>.*)$")
}
}
fn go_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str)> {
let (input, _) = tag("{\"time\":\"")(input)?;
let (input, time) = take_until("\"")(input)?;
let (input, _) = tag("\",\"level\":\"")(input)?;
let (input, level) = take_until("\"")(input)?;
let (input, _) = tag("\",\"msg\":\"")(input)?;
let (input, msg) = take_until("\"")(input)?;
let (input, _) = char('"')(input)?;
Ok((input, (time, level, msg)))
}
pub fn is_go_log_line(text: &str) -> bool {
match go_log_prefix(text) {
Ok((rest, _)) => rest.trim_end().ends_with('}'),
Err(_) => false,
}
}
fn py_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str, &str)> {
let (input, date) = take_while1(|c: char| c.is_ascii_digit() || c == '-')(input)?;
let (input, _) = char(' ')(input)?;
let (input, time) = take_while1(|c: char| c.is_ascii_digit() || c == ':' || c == ',')(input)?;
let (input, _) = char(' ')(input)?;
let (input, level) = take_while1(|c: char| c.is_ascii_uppercase())(input)?;
let (input, _) = char(' ')(input)?;
let (input, logger) =
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')(input)?;
let (input, _) = char(' ')(input)?;
Ok((input, (date, time, level, logger)))
}
pub fn is_py_log_line(text: &str) -> bool {
match py_log_prefix(text) {
Ok((rest, _)) => !extract_pairs(rest).is_empty(),
Err(_) => false,
}
}
pub fn detect_format(text: &str) -> &'static str {
if is_log_stream_line(text) {
"log_stream"
} else if is_go_log_line(text) {
"go-log"
} else if is_py_log_line(text) {
"py-log"
} else if serde_json::from_str::<serde_json::Value>(text).is_ok_and(|value| value.is_object()) {
"json"
} else if !extract_pairs(text).is_empty() {
"logfmt"
} else {
"none"
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_quoted_value() {
let pairs = extract_pairs(r#"msg="entity_moved""#);
assert_eq!(pairs, vec![("msg", "entity_moved", true)]);
}
#[test]
fn extracts_unquoted_bareword_value() {
let pairs = extract_pairs("x1=-4");
assert_eq!(pairs, vec![("x1", "-4", false)]);
}
#[test]
fn extracts_mixed_quoted_and_unquoted_pairs_in_one_line() {
let pairs = extract_pairs(r#"ts=1788648105773 msg="entity_moved" x1=-4 y1=34"#);
assert_eq!(
pairs,
vec![
("ts", "1788648105773", false),
("msg", "entity_moved", true),
("x1", "-4", false),
("y1", "34", false),
]
);
}
#[test]
fn returns_empty_for_lines_with_no_pairs() {
assert!(extract_pairs("just some plain prose here").is_empty());
assert!(extract_pairs(r#"{"foo": {"bar": 42}}"#).is_empty());
}
#[test]
fn does_not_panic_on_empty_value_at_end_of_line() {
assert_eq!(extract_pairs("key="), Vec::<(&str, &str, bool)>::new());
assert_eq!(extract_pairs(r#"key="""#), vec![("key", "", true)]);
}
#[test]
fn keeps_escaped_quote_inside_quoted_value() {
let pairs = extract_pairs(r#"msg="he said \"hi\"""#);
assert_eq!(pairs, vec![("msg", r#"he said \"hi\""#, true)]);
}
#[test]
fn detects_json_object_lines_as_json() {
assert_eq!(detect_format(r#"{"level":"error","code":500}"#), "json");
}
#[test]
fn detects_key_value_lines_as_logfmt() {
assert_eq!(detect_format(r#"ts=1788648105773 msg="entity_moved" x1=-4"#), "logfmt");
}
#[test]
fn detects_plain_prose_as_none() {
assert_eq!(detect_format("just some plain prose here"), "none");
}
#[test]
fn a_bare_json_array_or_scalar_is_not_treated_as_json() {
assert_eq!(detect_format("[1,2,3]"), "none");
assert_eq!(detect_format("42"), "none");
}
#[test]
fn rejects_base64_padding_fragment_as_a_pair() {
let pairs = extract_pairs(
"qname: <mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='>, qtype: A, client pid: 12260 (firefox)",
);
assert!(
pairs.iter().all(|(k, _, _)| *k != "rKTEtG7JmnThhoKPbZiVrA"),
"base64 padding fragment must not be extracted as a pair: {pairs:?}"
);
}
const LOG_STREAM_FIXTURE: &str = include_str!("../fixtures/log_stream_sample.txt");
#[test]
fn detects_log_stream_shaped_lines() {
let data_line = LOG_STREAM_FIXTURE.lines().nth(1).unwrap();
assert_eq!(detect_format(data_line), "log_stream");
assert!(is_log_stream_line(data_line));
}
#[test]
fn header_row_is_not_detected_as_log_stream() {
let header = LOG_STREAM_FIXTURE.lines().next().unwrap();
assert!(!is_log_stream_line(header));
}
fn column_names(header: &LogStreamHeader) -> Vec<&str> {
header.columns.iter().map(|c| c.name.as_str()).collect()
}
#[test]
fn detects_log_stream_header_columns_generically() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).expect("fixture header must be detected");
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type", "Activity", "PID", "TTL"]);
assert!(!header.tab_delimited);
}
#[test]
fn a_log_stream_data_line_is_never_detected_as_its_own_header() {
let data_line = LOG_STREAM_FIXTURE.lines().nth(1).unwrap();
assert!(LogStreamHeader::detect(data_line).is_none());
}
#[test]
fn plain_prose_is_not_detected_as_a_log_stream_header() {
assert!(LogStreamHeader::detect("just some plain prose here").is_none());
assert!(LogStreamHeader::detect("ts=1788648105773 msg=\"entity_moved\"").is_none());
}
#[test]
fn splits_log_stream_row_into_named_columns_plus_trailing_message() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let data_line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let row = header.split_row(data_line);
assert_eq!(row.len(), header.columns.len() + 1);
assert_eq!(row[0], "2026-09-06 01:48:54.518117+0100");
assert_eq!(row[1], "0x4dfed2");
assert_eq!(row[2], "Default");
assert_eq!(row[3], "0x0");
assert_eq!(row[4], "606");
assert_eq!(row[5], "0");
assert_eq!(row[6], "WindowServer: (HSTouchHIDService) something happened");
}
#[test]
fn a_single_tab_separates_columns_exactly_like_a_run_of_spaces() {
let header = LogStreamHeader::detect("Timestamp\tThread\tType").unwrap();
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type"]);
assert!(header.tab_delimited);
let spaced = header.split_row("2026-09-06 01:48:54\t0x4dfed2\t\tDefault trailing message");
let tabbed = header.split_row("2026-09-06 01:48:54\t0x4dfed2\tDefault trailing message");
assert_eq!(spaced[0], "2026-09-06 01:48:54");
assert_eq!(spaced[1], "0x4dfed2");
assert_eq!(spaced[2], "Default trailing message");
assert_eq!(tabbed[0], "2026-09-06 01:48:54");
assert_eq!(tabbed[1], "0x4dfed2");
assert_eq!(tabbed[2], "Default trailing message");
}
#[test]
fn column_and_message_patterns_capture_the_same_fields_split_log_stream_row_finds() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let type_re = Regex::new(&header.column_pattern(2, "Type")).unwrap();
assert_eq!(type_re.captures(line).unwrap().name("Type").unwrap().as_str(), "Default");
let pid_re = Regex::new(&header.column_pattern(4, "PID")).unwrap();
assert_eq!(pid_re.captures(line).unwrap().name("PID").unwrap().as_str(), "606");
let message_re = Regex::new(&header.message_pattern("message")).unwrap();
assert_eq!(
message_re.captures(line).unwrap().name("message").unwrap().as_str(),
"WindowServer: (HSTouchHIDService) something happened"
);
let timestamp_re = Regex::new(&header.column_pattern(0, "Timestamp")).unwrap();
assert_eq!(
timestamp_re.captures(line).unwrap().name("Timestamp").unwrap().as_str(),
"2026-09-06 01:48:54.518117+0100"
);
}
#[test]
fn is_log_stream_line_rejects_non_columnar_lines_that_a_fixed_offset_column_regex_would_still_match() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let timestamp_re = Regex::new(&header.column_pattern(0, "Timestamp")).unwrap();
let continuation_line = "so_gencnt: 4042707 so_state: 0x0";
assert!(timestamp_re.captures(continuation_line).is_some(), "sanity: regex alone still matches garbage");
assert!(!is_log_stream_line(continuation_line));
assert!(!is_log_stream_line(header_line));
assert!(!is_log_stream_line("}"));
}
#[test]
fn a_wifi_stat_dump_line_is_still_recognized_as_log_stream_despite_embedded_key_value_pairs() {
let line = LOG_STREAM_FIXTURE
.lines()
.find(|line| line.contains("txRTSFrm="))
.expect("fixture must contain the WiFi stat dump line");
assert_eq!(detect_format(line), "log_stream");
assert!(!extract_pairs(line).is_empty());
}
const GO_LOG_FIXTURE: &str = include_str!("../fixtures/go_log_sample.txt");
const PY_LOG_FIXTURE: &str = include_str!("../fixtures/py_log_sample.txt");
#[test]
fn detects_go_log_shaped_lines() {
for line in GO_LOG_FIXTURE.lines() {
assert!(is_go_log_line(line), "expected go-log line to match: {line}");
assert_eq!(detect_format(line), "go-log");
}
}
#[test]
fn detects_py_log_shaped_lines() {
for line in PY_LOG_FIXTURE.lines() {
assert!(is_py_log_line(line), "expected py-log line to match: {line}");
assert_eq!(detect_format(line), "py-log");
}
}
#[test]
fn a_generic_json_object_is_not_detected_as_go_log() {
let line = r#"{"level":"error","code":500}"#;
assert!(!is_go_log_line(line));
assert_eq!(detect_format(line), "json");
}
#[test]
fn a_generic_logfmt_line_is_not_detected_as_py_log() {
let line = r#"ts=1788648105773 msg="entity_moved" x1=-4"#;
assert!(!is_py_log_line(line));
assert_eq!(detect_format(line), "logfmt");
}
#[test]
fn go_log_prefix_extracts_time_level_and_msg() {
let (rest, (time, level, msg)) =
go_log_prefix(r#"{"time":"2026-09-06T10:35:34.499921+01:00","level":"WARN","msg":"eventC","labelC":1}"#)
.unwrap();
assert_eq!(time, "2026-09-06T10:35:34.499921+01:00");
assert_eq!(level, "WARN");
assert_eq!(msg, "eventC");
assert_eq!(rest, r#","labelC":1}"#);
}
#[test]
fn py_log_prefix_extracts_date_time_level_and_logger() {
let (rest, (date, time, level, logger)) =
py_log_prefix("2026-09-06 10:35:55,217 INFO delete-py-log event=eventC labelC=1").unwrap();
assert_eq!(date, "2026-09-06");
assert_eq!(time, "10:35:55,217");
assert_eq!(level, "INFO");
assert_eq!(logger, "delete-py-log");
assert_eq!(rest, "event=eventC labelC=1");
}
}
src/engine/message.rs
+84
View File
@@ -0,0 +1,84 @@
//! The engine actor's inbound message/query types and its outbound response type.
use crate::api::dto::{
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
PromqlSeriesDto, StatsDto, TimelineBucketDto, TimelineParams,
};
use crate::engine::{ClearParamsResult, DeclareMetricRequest, DeclareParamRequest, IngestedLine, MetricSample};
use std::collections::BTreeMap;
use tokio::sync::oneshot;
#[allow(dead_code)]
pub enum EngineQuery {
Logs(LogsParams),
Timeline(TimelineParams),
Stats,
ParamDeclarations,
MetricDeclarations,
BucketValues {
metric_name: String,
labels_filter: Option<BTreeMap<String, String>>,
start_ms: Option<i64>,
end_ms: Option<i64>,
since_sequence: Option<u64>,
limit: Option<usize>,
},
Promql { query: String, start_ms: Option<i64>, end_ms: Option<i64> },
}
#[allow(dead_code)]
pub enum EngineMessage {
Ingest(IngestedLine),
IngestMetricSample(MetricSample),
DeclareParam {
request: DeclareParamRequest,
respond_to: oneshot::Sender<Result<(), String>>,
},
RemoveParamDeclaration {
name: String,
respond_to: oneshot::Sender<bool>,
},
ClearParams {
respond_to: oneshot::Sender<ClearParamsResult>,
},
DeclareMetric {
request: DeclareMetricRequest,
respond_to: oneshot::Sender<Result<(), String>>,
},
UpdateMetric {
name: String,
request: DeclareMetricRequest,
respond_to: oneshot::Sender<Result<(), String>>,
},
RemoveMetricDeclaration {
name: String,
respond_to: oneshot::Sender<bool>,
},
SetMaxLogs(Option<usize>),
EnsureSortHelperIndex {
param_name: String,
respond_to: oneshot::Sender<()>,
},
Barrier(oneshot::Sender<()>),
}
#[allow(dead_code)]
pub enum EngineResponse {
Logs(Vec<LogEntryDto>),
Timeline(Vec<TimelineBucketDto>),
Stats(StatsDto),
ParamDeclarations(Vec<ParamDeclarationDto>),
MetricDeclarations(Vec<MetricDeclarationDto>),
BucketValues(Vec<BucketValueDto>),
Promql(Result<Vec<PromqlSeriesDto>, String>),
Error(String),
}
impl EngineQuery {
pub(crate) fn sort_by(&self) -> Option<&str> {
match self {
EngineQuery::Logs(params) => params.sort_by.as_deref(),
_ => None,
}
}
}
src/metric_primitive.rs → src/engine/metric_primitive.rs
+1 -7
View File
@@ -1,18 +1,15 @@
use crate::timestamp::Timestamp; use crate::engine::timestamp::Timestamp;
use serde::Serialize; use serde::Serialize;
use std::cmp::Ordering; use std::cmp::Ordering;
use std::fmt; use std::fmt;
/// A simple, self-describing value used by metrics/labels without pulling in JSON.
#[allow(dead_code)] #[allow(dead_code)]
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
pub enum MetricPrimitive { pub enum MetricPrimitive {
String(String), String(String),
Number(Number), Number(Number),
Timestamp(Timestamp), Timestamp(Timestamp),
/// Milliseconds elapsed; distinct from `Number` so it reads unambiguously in JSON/label form.
Duration(i64), Duration(i64),
/// A `Boolean`-kind param's value: whether `message_match_regex` matched the line. See `MetricPrimitiveKind::Boolean`.
Boolean(bool), Boolean(bool),
} }
@@ -51,7 +48,6 @@ impl Ord for MetricPrimitive {
} }
} }
/// Textual form used to compare label filters (query-string values) against a formatted `MetricPrimitive` rather than parsing them into one.
impl fmt::Display for MetricPrimitive { impl fmt::Display for MetricPrimitive {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self { match self {
@@ -65,7 +61,6 @@ impl fmt::Display for MetricPrimitive {
} }
} }
/// Numbers are kept as a tagged int/float pair so exact integers stay exact.
#[allow(dead_code)] #[allow(dead_code)]
#[derive(Debug, Clone, Copy, Serialize)] #[derive(Debug, Clone, Copy, Serialize)]
pub enum Number { pub enum Number {
@@ -74,7 +69,6 @@ pub enum Number {
} }
impl Number { impl Number {
/// Widens to `f64` for cross-kind numeric comparisons; `pub(crate)` since the filter evaluator in `engine.rs` needs it.
pub(crate) fn as_f64(&self) -> f64 { pub(crate) fn as_f64(&self) -> f64 {
match self { match self {
Number::Int(v) => *v as f64, Number::Int(v) => *v as f64,
src/engine/mod.rs
+64
View File
@@ -0,0 +1,64 @@
pub mod alerts;
pub mod data_value;
pub mod filter_datetime;
pub mod filter_parser;
pub mod handle;
pub mod index_key;
pub mod ingest;
pub mod logfmt;
pub mod message;
pub mod metric_primitive;
pub mod promql;
pub mod rate;
pub mod snapshot;
pub mod timestamp;
#[cfg(test)]
mod tests;
pub use handle::{spawn, EngineHandle};
#[cfg(test)]
pub(crate) use handle::test_handle;
pub use message::{EngineMessage, EngineQuery, EngineResponse};
pub(crate) use snapshot::{eval_json_comparison, EngineSnapshot};
use crate::engine::data_value::ParamDeclaration;
use crate::engine::timestamp::Timestamp;
use serde::Deserialize;
use std::collections::BTreeMap;
pub struct IngestedLine {
pub sequence: u64,
pub timestamp: Timestamp,
pub text: String,
}
pub struct MetricSample {
pub metric_name: String,
pub labels: BTreeMap<String, String>,
pub value: f64,
pub timestamp: Timestamp,
}
pub type DeclareParamRequest = ParamDeclaration;
#[derive(Debug, Clone, serde::Serialize)]
pub struct ClearParamsResult {
pub removed: usize,
pub reextracted: usize,
pub logs_considered: usize,
}
#[derive(Debug, Clone, Deserialize)]
pub struct DeclareMetricRequest {
pub name: String,
pub message_match_regex: String,
#[serde(default)]
pub label_params: Vec<String>,
#[serde(default)]
pub value_param: Option<String>,
#[serde(default)]
pub matched_labels: Option<BTreeMap<String, String>>,
#[serde(default)]
pub unmatched_labels: Option<BTreeMap<String, String>>,
}
src/promql.rs → src/engine/promql.rs
+79 -50
View File
@@ -1,17 +1,12 @@
//! Hand-rolled parser for a small PromQL subset: metric name + label matchers, an optional `[5m]` range, and a top-level `sum()`/`avg()`/`rate()`/`histogram()`/`gauge()` wrapper.
//! Only exact-match (`=`) label matchers are supported; `rate()` requires a duration range selector, while `histogram(metric[width])` reuses that `[...]` slot for a bare bucket width instead.
//! `sum()`/`avg()` may additionally wrap a `rate(...)` call directly (e.g. `sum(rate(requests[5m]))`), and either may be followed by a `by (label, ...)` clause to preserve one series per distinct value of the named labels instead of collapsing to a single series.
use std::time::Duration; use std::time::Duration;
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AggOp { pub enum AggOp {
Sum, Sum,
Avg, Avg,
Count,
Rate, Rate,
/// Buckets counter values into fixed-width buckets; requires `bucket_width`.
Histogram, Histogram,
/// The most-recent counter value per label combination.
Gauge, Gauge,
} }
@@ -26,26 +21,19 @@ pub struct PromqlQuery {
pub agg: Option<AggOp>, pub agg: Option<AggOp>,
pub metric_name: String, pub metric_name: String,
pub matchers: Vec<LabelMatcher>, pub matchers: Vec<LabelMatcher>,
/// Duration range selector, e.g. `[5m]`; only meaningful for `rate()`.
pub range: Option<Duration>, pub range: Option<Duration>,
/// Bare-number range selector, e.g. `[10]`; only meaningful for `histogram()`, where it's the bucket width.
pub bucket_width: Option<f64>, pub bucket_width: Option<f64>,
/// Set when `agg` is `Sum`/`Avg` and it wraps an inner `rate(metric[range])` rather than a bare vector
/// selector, e.g. `sum(rate(requests[5m]))`. `range` above carries the inner `rate()`'s duration in that case.
pub inner_rate: bool, pub inner_rate: bool,
/// Labels named in a trailing `by (...)` clause; empty means "collapse to a single series" (the historical
/// behavior of `sum()`/`avg()`). When non-empty, one series is kept per distinct combination of these labels.
pub by_labels: Vec<String>, pub by_labels: Vec<String>,
} }
/// What a `[...]` range selector turned out to hold: a duration, or a bare width.
enum RangeSelector { enum RangeSelector {
Duration(Duration), Duration(Duration),
BareWidth(f64), BareWidth(f64),
} }
/// Parses a PromQL query string, returning a human-readable error message on malformed input. impl PromqlQuery {
pub fn parse_query(input: &str) -> Result<PromqlQuery, String> { pub fn parse(input: &str) -> Result<PromqlQuery, String> {
let mut parser = Parser::new(input); let mut parser = Parser::new(input);
let query = parser.parse_query()?; let query = parser.parse_query()?;
parser.skip_whitespace(); parser.skip_whitespace();
@@ -61,6 +49,7 @@ pub fn parse_query(input: &str) -> Result<PromqlQuery, String> {
); );
} }
Ok(query) Ok(query)
}
} }
struct Parser<'a> { struct Parser<'a> {
@@ -173,7 +162,6 @@ impl<'a> Parser<'a> {
Ok(matchers) Ok(matchers)
} }
/// Parses `[<digits><unit>]` (a duration, e.g. `[5m]`) or a bare `[<digits>]` (a bucket width).
fn parse_range_selector(&mut self) -> Result<Option<RangeSelector>, String> { fn parse_range_selector(&mut self) -> Result<Option<RangeSelector>, String> {
self.skip_whitespace(); self.skip_whitespace();
if self.peek() != Some('[') { if self.peek() != Some('[') {
@@ -235,7 +223,6 @@ impl<'a> Parser<'a> {
}) })
} }
/// Parses a trailing `by (label, ...)` clause, if present; returns an empty vec otherwise.
fn parse_by_clause(&mut self) -> Result<Vec<String>, String> { fn parse_by_clause(&mut self) -> Result<Vec<String>, String> {
self.skip_whitespace(); self.skip_whitespace();
let checkpoint = self.pos; let checkpoint = self.pos;
@@ -278,16 +265,18 @@ impl<'a> Parser<'a> {
let agg = match ident.as_str() { let agg = match ident.as_str() {
"sum" => Some(AggOp::Sum), "sum" => Some(AggOp::Sum),
"avg" => Some(AggOp::Avg), "avg" => Some(AggOp::Avg),
"count" => Some(AggOp::Count),
"rate" => Some(AggOp::Rate), "rate" => Some(AggOp::Rate),
"histogram" => Some(AggOp::Histogram), "histogram" => Some(AggOp::Histogram),
"gauge" => Some(AggOp::Gauge), "gauge" => Some(AggOp::Gauge),
_ => None, _ => None,
}; };
self.skip_whitespace(); self.skip_whitespace();
let prefix_by_labels = if agg.is_some() { self.parse_by_clause()? } else { Vec::new() };
self.skip_whitespace();
if agg.is_some() && self.peek() == Some('(') { if agg.is_some() && self.peek() == Some('(') {
self.expect_char('(')?; self.expect_char('(')?;
self.skip_whitespace(); self.skip_whitespace();
// `sum()`/`avg()` may wrap `rate(metric[range])` directly instead of a bare vector selector.
let mut inner = if matches!(agg, Some(AggOp::Sum) | Some(AggOp::Avg)) { let mut inner = if matches!(agg, Some(AggOp::Sum) | Some(AggOp::Avg)) {
let inner_checkpoint = self.pos; let inner_checkpoint = self.pos;
let took_inner_rate = self.parse_identifier().is_ok_and(|id| id == "rate"); let took_inner_rate = self.parse_identifier().is_ok_and(|id| id == "rate");
@@ -318,10 +307,17 @@ impl<'a> Parser<'a> {
}; };
self.expect_char(')')?; self.expect_char(')')?;
inner.agg = agg; inner.agg = agg;
inner.by_labels = self.parse_by_clause()?; let trailing_by_labels = self.parse_by_clause()?;
if !prefix_by_labels.is_empty() && !trailing_by_labels.is_empty() {
return Err(
"by (...) clause may only appear once, either before or after the aggregation's argument"
.to_string(),
);
}
inner.by_labels =
if prefix_by_labels.is_empty() { trailing_by_labels } else { prefix_by_labels };
Ok(inner) Ok(inner)
} else { } else {
// Rewind and parse as a plain vector selector, covering metric names that collide with an agg_op keyword.
self.pos = checkpoint; self.pos = checkpoint;
self.parse_vector_selector() self.parse_vector_selector()
} }
@@ -334,7 +330,7 @@ mod tests {
#[test] #[test]
fn parses_bare_metric_name() { fn parses_bare_metric_name() {
let query = parse_query("requests").unwrap(); let query = PromqlQuery::parse("requests").unwrap();
assert_eq!(query.agg, None); assert_eq!(query.agg, None);
assert_eq!(query.metric_name, "requests"); assert_eq!(query.metric_name, "requests");
assert!(query.matchers.is_empty()); assert!(query.matchers.is_empty());
@@ -343,7 +339,7 @@ mod tests {
#[test] #[test]
fn parses_metric_with_single_label_matcher() { fn parses_metric_with_single_label_matcher() {
let query = parse_query(r#"requests{path="/a"}"#).unwrap(); let query = PromqlQuery::parse(r#"requests{path="/a"}"#).unwrap();
assert_eq!(query.metric_name, "requests"); assert_eq!(query.metric_name, "requests");
assert_eq!( assert_eq!(
query.matchers, query.matchers,
@@ -353,7 +349,7 @@ mod tests {
#[test] #[test]
fn parses_metric_with_multiple_label_matchers() { fn parses_metric_with_multiple_label_matchers() {
let query = parse_query(r#"requests{path="/a", method="GET"}"#).unwrap(); let query = PromqlQuery::parse(r#"requests{path="/a", method="GET"}"#).unwrap();
assert_eq!(query.matchers.len(), 2); assert_eq!(query.matchers.len(), 2);
assert_eq!(query.matchers[0].name, "path"); assert_eq!(query.matchers[0].name, "path");
assert_eq!(query.matchers[1].name, "method"); assert_eq!(query.matchers[1].name, "method");
@@ -361,91 +357,105 @@ mod tests {
#[test] #[test]
fn parses_range_selector() { fn parses_range_selector() {
let query = parse_query("requests[5m]").unwrap(); let query = PromqlQuery::parse("requests[5m]").unwrap();
assert_eq!(query.range, Some(Duration::from_secs(300))); assert_eq!(query.range, Some(Duration::from_secs(300)));
} }
#[test] #[test]
fn parses_label_matchers_and_range_together() { fn parses_label_matchers_and_range_together() {
let query = parse_query(r#"requests{path="/a"}[1h]"#).unwrap(); let query = PromqlQuery::parse(r#"requests{path="/a"}[1h]"#).unwrap();
assert_eq!(query.matchers.len(), 1); assert_eq!(query.matchers.len(), 1);
assert_eq!(query.range, Some(Duration::from_secs(3600))); assert_eq!(query.range, Some(Duration::from_secs(3600)));
} }
#[test] #[test]
fn parses_sum_aggregation() { fn parses_sum_aggregation() {
let query = parse_query(r#"sum(requests{path="/a"})"#).unwrap(); let query = PromqlQuery::parse(r#"sum(requests{path="/a"})"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Sum)); assert_eq!(query.agg, Some(AggOp::Sum));
assert_eq!(query.metric_name, "requests"); assert_eq!(query.metric_name, "requests");
} }
#[test] #[test]
fn parses_avg_aggregation() { fn parses_avg_aggregation() {
let query = parse_query("avg(requests)").unwrap(); let query = PromqlQuery::parse("avg(requests)").unwrap();
assert_eq!(query.agg, Some(AggOp::Avg)); assert_eq!(query.agg, Some(AggOp::Avg));
} }
#[test] #[test]
fn parses_rate_with_range() { fn parses_rate_with_range() {
let query = parse_query("rate(requests[5m])").unwrap(); let query = PromqlQuery::parse("rate(requests[5m])").unwrap();
assert_eq!(query.agg, Some(AggOp::Rate)); assert_eq!(query.agg, Some(AggOp::Rate));
assert_eq!(query.range, Some(Duration::from_secs(300))); assert_eq!(query.range, Some(Duration::from_secs(300)));
} }
#[test] #[test]
fn rejects_rate_without_range() { fn rejects_rate_without_range() {
let err = parse_query("rate(requests)").unwrap_err(); let err = PromqlQuery::parse("rate(requests)").unwrap_err();
assert!(err.contains("range selector")); assert!(err.contains("range selector"));
} }
#[test] #[test]
fn rejects_empty_input() { fn rejects_empty_input() {
assert!(parse_query("").is_err()); assert!(PromqlQuery::parse("").is_err());
} }
#[test] #[test]
fn rejects_unclosed_brace() { fn rejects_unclosed_brace() {
assert!(parse_query(r#"requests{path="/a""#).is_err()); assert!(PromqlQuery::parse(r#"requests{path="/a""#).is_err());
} }
#[test] #[test]
fn rejects_unclosed_bracket() { fn rejects_unclosed_bracket() {
assert!(parse_query("requests[5m").is_err()); assert!(PromqlQuery::parse("requests[5m").is_err());
} }
#[test] #[test]
fn rejects_unknown_aggregation() { fn rejects_unknown_aggregation() {
// `count` isn't a supported agg_op, so this parses as a bare metric name and trips on the trailing `(...)`. let err = PromqlQuery::parse("median(requests)").unwrap_err();
let err = parse_query("count(requests)").unwrap_err();
assert!(err.contains("unexpected trailing input")); assert!(err.contains("unexpected trailing input"));
} }
#[test]
fn parses_count_aggregation() {
let query = PromqlQuery::parse("count(requests)").unwrap();
assert_eq!(query.agg, Some(AggOp::Count));
assert_eq!(query.metric_name, "requests");
assert!(!query.inner_rate);
}
#[test]
fn parses_count_with_by_clause() {
let query = PromqlQuery::parse("count(requests) by (path)").unwrap();
assert_eq!(query.agg, Some(AggOp::Count));
assert_eq!(query.by_labels, vec!["path".to_string()]);
}
#[test] #[test]
fn rejects_regex_label_matcher() { fn rejects_regex_label_matcher() {
let err = parse_query(r#"requests{path=~"/a"}"#).unwrap_err(); let err = PromqlQuery::parse(r#"requests{path=~"/a"}"#).unwrap_err();
assert!(err.contains("regex label matchers")); assert!(err.contains("regex label matchers"));
} }
#[test] #[test]
fn rejects_missing_metric_name() { fn rejects_missing_metric_name() {
assert!(parse_query(r#"{path="/a"}"#).is_err()); assert!(PromqlQuery::parse(r#"{path="/a"}"#).is_err());
} }
#[test] #[test]
fn rejects_unknown_duration_unit() { fn rejects_unknown_duration_unit() {
let err = parse_query("requests[5x]").unwrap_err(); let err = PromqlQuery::parse("requests[5x]").unwrap_err();
assert!(err.contains("duration unit")); assert!(err.contains("duration unit"));
} }
#[test] #[test]
fn rejects_trailing_garbage() { fn rejects_trailing_garbage() {
let err = parse_query("requests extra").unwrap_err(); let err = PromqlQuery::parse("requests extra").unwrap_err();
assert!(err.contains("unexpected trailing input")); assert!(err.contains("unexpected trailing input"));
} }
#[test] #[test]
fn parses_histogram_with_bare_width() { fn parses_histogram_with_bare_width() {
let query = parse_query("histogram(latency_ms[10])").unwrap(); let query = PromqlQuery::parse("histogram(latency_ms[10])").unwrap();
assert_eq!(query.agg, Some(AggOp::Histogram)); assert_eq!(query.agg, Some(AggOp::Histogram));
assert_eq!(query.metric_name, "latency_ms"); assert_eq!(query.metric_name, "latency_ms");
assert_eq!(query.bucket_width, Some(10.0)); assert_eq!(query.bucket_width, Some(10.0));
@@ -454,27 +464,27 @@ mod tests {
#[test] #[test]
fn rejects_histogram_without_width() { fn rejects_histogram_without_width() {
let err = parse_query("histogram(latency_ms)").unwrap_err(); let err = PromqlQuery::parse("histogram(latency_ms)").unwrap_err();
assert!(err.contains("bucket width")); assert!(err.contains("bucket width"));
} }
#[test] #[test]
fn parses_gauge_with_no_range_selector() { fn parses_gauge_with_no_range_selector() {
let query = parse_query(r#"gauge(connections{host="a"})"#).unwrap(); let query = PromqlQuery::parse(r#"gauge(connections{host="a"})"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Gauge)); assert_eq!(query.agg, Some(AggOp::Gauge));
assert_eq!(query.matchers, vec![LabelMatcher { name: "host".to_string(), value: "a".to_string() }]); assert_eq!(query.matchers, vec![LabelMatcher { name: "host".to_string(), value: "a".to_string() }]);
} }
#[test] #[test]
fn rate_range_selector_is_still_a_duration_not_a_bare_width() { fn rate_range_selector_is_still_a_duration_not_a_bare_width() {
let query = parse_query("rate(requests[5m])").unwrap(); let query = PromqlQuery::parse("rate(requests[5m])").unwrap();
assert_eq!(query.range, Some(Duration::from_secs(300))); assert_eq!(query.range, Some(Duration::from_secs(300)));
assert_eq!(query.bucket_width, None); assert_eq!(query.bucket_width, None);
} }
#[test] #[test]
fn parses_sum_of_rate() { fn parses_sum_of_rate() {
let query = parse_query("sum(rate(requests[1m]))").unwrap(); let query = PromqlQuery::parse("sum(rate(requests[1m]))").unwrap();
assert_eq!(query.agg, Some(AggOp::Sum)); assert_eq!(query.agg, Some(AggOp::Sum));
assert!(query.inner_rate); assert!(query.inner_rate);
assert_eq!(query.metric_name, "requests"); assert_eq!(query.metric_name, "requests");
@@ -484,7 +494,7 @@ mod tests {
#[test] #[test]
fn parses_avg_of_rate_with_label_matchers() { fn parses_avg_of_rate_with_label_matchers() {
let query = parse_query(r#"avg(rate(requests{path="/a"}[5m]))"#).unwrap(); let query = PromqlQuery::parse(r#"avg(rate(requests{path="/a"}[5m]))"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Avg)); assert_eq!(query.agg, Some(AggOp::Avg));
assert!(query.inner_rate); assert!(query.inner_rate);
assert_eq!(query.matchers, vec![LabelMatcher { name: "path".to_string(), value: "/a".to_string() }]); assert_eq!(query.matchers, vec![LabelMatcher { name: "path".to_string(), value: "/a".to_string() }]);
@@ -492,13 +502,13 @@ mod tests {
#[test] #[test]
fn rejects_sum_of_rate_without_range() { fn rejects_sum_of_rate_without_range() {
let err = parse_query("sum(rate(requests))").unwrap_err(); let err = PromqlQuery::parse("sum(rate(requests))").unwrap_err();
assert!(err.contains("range selector")); assert!(err.contains("range selector"));
} }
#[test] #[test]
fn parses_sum_with_by_clause() { fn parses_sum_with_by_clause() {
let query = parse_query("sum(requests) by (path)").unwrap(); let query = PromqlQuery::parse("sum(requests) by (path)").unwrap();
assert_eq!(query.agg, Some(AggOp::Sum)); assert_eq!(query.agg, Some(AggOp::Sum));
assert!(!query.inner_rate); assert!(!query.inner_rate);
assert_eq!(query.by_labels, vec!["path".to_string()]); assert_eq!(query.by_labels, vec!["path".to_string()]);
@@ -506,25 +516,44 @@ mod tests {
#[test] #[test]
fn parses_sum_of_rate_with_by_clause_multiple_labels() { fn parses_sum_of_rate_with_by_clause_multiple_labels() {
let query = parse_query("sum(rate(requests[1m])) by (path, method)").unwrap(); let query = PromqlQuery::parse("sum(rate(requests[1m])) by (path, method)").unwrap();
assert!(query.inner_rate); assert!(query.inner_rate);
assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]); assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]);
} }
#[test] #[test]
fn sum_without_by_clause_has_empty_by_labels() { fn sum_without_by_clause_has_empty_by_labels() {
let query = parse_query("sum(requests)").unwrap(); let query = PromqlQuery::parse("sum(requests)").unwrap();
assert!(query.by_labels.is_empty()); assert!(query.by_labels.is_empty());
} }
#[test] #[test]
fn rejects_empty_by_clause() { fn rejects_empty_by_clause() {
let err = parse_query("sum(requests) by ()").unwrap_err(); let err = PromqlQuery::parse("sum(requests) by ()").unwrap_err();
assert!(err.contains("at least one label")); assert!(err.contains("at least one label"));
} }
#[test] #[test]
fn rejects_unterminated_by_clause() { fn rejects_unterminated_by_clause() {
assert!(parse_query("sum(requests) by (path").is_err()); assert!(PromqlQuery::parse("sum(requests) by (path").is_err());
}
#[test]
fn parses_sum_with_prefix_by_clause() {
let query = PromqlQuery::parse("sum by (path) (requests)").unwrap();
assert_eq!(query.agg, Some(AggOp::Sum));
assert_eq!(query.by_labels, vec!["path".to_string()]);
}
#[test]
fn parses_sum_with_prefix_by_clause_multiple_labels() {
let query = PromqlQuery::parse("sum by (path, method) (requests)").unwrap();
assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]);
}
#[test]
fn rejects_by_clause_both_before_and_after_aggregation() {
let err = PromqlQuery::parse("sum by (path) (requests) by (method)").unwrap_err();
assert!(err.contains("only appear once"));
} }
} }
src/engine/rate.rs
+108
View File
@@ -0,0 +1,108 @@
use std::collections::VecDeque;
use crate::engine::timestamp::Timestamp;
fn count_in_window(points: &[(Timestamp, f64)], now: Timestamp, window_ms: i64) -> usize {
let now_ms = now.as_millis();
let window_start = now_ms - window_ms;
points
.iter()
.filter(|(ts, _)| {
let ms = ts.as_millis();
ms > window_start && ms <= now_ms
})
.count()
}
pub fn rate_at(points: &[(Timestamp, f64)], now: Timestamp, window_ms: i64) -> f64 {
let window_seconds = window_ms as f64 / 1000.0;
count_in_window(points, now, window_ms) as f64 / window_seconds
}
pub fn windowed_rate_series(points: &[(Timestamp, f64)], window_ms: i64) -> Vec<(Timestamp, f64)> {
points.iter().map(|(ts, _)| (*ts, rate_at(points, *ts, window_ms))).collect()
}
#[derive(Debug, Default, Clone)]
pub struct RateWindow {
timestamps: VecDeque<Timestamp>,
}
impl RateWindow {
pub fn backfill(timestamps: impl IntoIterator<Item = Timestamp>, now: Timestamp, window_ms: i64) -> Self {
let mut window = Self::default();
for ts in timestamps {
window.push(ts, window_ms);
}
window.prune(now, window_ms);
window
}
pub fn push(&mut self, timestamp: Timestamp, window_ms: i64) {
self.timestamps.push_back(timestamp);
self.prune(timestamp, window_ms);
}
fn prune(&mut self, now: Timestamp, window_ms: i64) {
let window_start = now.as_millis() - window_ms;
while let Some(&front) = self.timestamps.front() {
if front.as_millis() <= window_start {
self.timestamps.pop_front();
} else {
break;
}
}
}
pub fn count(&self) -> usize {
self.timestamps.len()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn rate_at_matches_hand_computed_window() {
let points = vec![
(Timestamp::from_millis(0), 1.0),
(Timestamp::from_millis(1000), 1.0),
(Timestamp::from_millis(2000), 1.0),
];
assert_eq!(rate_at(&points, Timestamp::from_millis(2000), 1000), 1.0);
assert_eq!(rate_at(&points, Timestamp::from_millis(2000), 2000), 1.0);
}
#[test]
fn windowed_rate_series_matches_rate_at_at_each_point() {
let points = vec![
(Timestamp::from_millis(0), 1.0),
(Timestamp::from_millis(500), 1.0),
(Timestamp::from_millis(1000), 1.0),
];
let series = windowed_rate_series(&points, 1000);
for (ts, rate) in &series {
assert_eq!(*rate, rate_at(&points, *ts, 1000));
}
}
#[test]
fn rate_window_incremental_count_matches_batch_count_in_window() {
let mut window = RateWindow::default();
let millis = [0, 200, 400, 1200];
for ms in millis {
window.push(Timestamp::from_millis(ms), 1000);
}
let points: Vec<(Timestamp, f64)> =
millis.iter().map(|ms| (Timestamp::from_millis(*ms), 1.0)).collect();
assert_eq!(window.count(), count_in_window(&points, Timestamp::from_millis(1200), 1000));
}
#[test]
fn rate_window_backfill_seeds_from_history_within_the_window() {
let history = [0, 500, 2000, 2500].map(Timestamp::from_millis);
let window = RateWindow::backfill(history, Timestamp::from_millis(2500), 1000);
assert_eq!(window.count(), 2);
}
}
src/engine/snapshot/filter.rs
+261
View File
@@ -0,0 +1,261 @@
//! Compiling and evaluating the search-bar filter syntax (`filter_parser`) against log lines
//! and buckets: regex matches, the `ottyl_seq`/`ottyl_ts`/`ottyl_fmt` pseudo-metrics, per-param
//! metric-bucket comparisons, and ad-hoc JSON field comparisons.
use crate::engine::filter_datetime;
use crate::engine::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::engine::index_key::IndexKey;
use crate::engine::data_value::DataValue;
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::timestamp::Timestamp;
use regex::Regex;
use std::cmp::Ordering;
use std::collections::HashSet;
pub(super) enum CompiledFilter {
Regex(Regex),
Sequence { op: CompareOp, target: f64 },
Timestamp { op: CompareOp, target_ms: i64 },
Fmt { op: CompareOp, target: String },
MetricMatch(HashSet<u64>),
Json { field: String, op: CompareOp, value: FilterValue },
And(Box<CompiledFilter>, Box<CompiledFilter>),
Or(Box<CompiledFilter>, Box<CompiledFilter>),
}
impl EngineSnapshot {
pub(super) fn compile_filter_query(&self, filter: &str, now: Timestamp) -> Result<CompiledFilter, String> {
let expr = filter_parser::parse(filter).map_err(|err| err.to_string())?;
self.compile_filter(&expr, now)
}
fn compile_filter(&self, expr: &FilterExpr, now: Timestamp) -> Result<CompiledFilter, String> {
match expr {
FilterExpr::Regex(pattern) => Regex::new(pattern)
.map(CompiledFilter::Regex)
.map_err(|err| format!("invalid regex in filter: {err}")),
FilterExpr::Comparison { metric, param, op, value } => match metric.as_str() {
"ottyl_seq" => CompiledFilter::from_seq_comparison(*op, value),
"ottyl_ts" => CompiledFilter::from_ts_comparison(*op, value, now),
"ottyl_fmt" => Ok(CompiledFilter::from_fmt_comparison(*op, value)),
other => self.compile_metric_comparison(other, param.as_deref(), *op, value, now),
},
FilterExpr::JsonComparison { field, op, value } => Ok(CompiledFilter::Json {
field: field.clone(),
op: *op,
value: value.clone(),
}),
FilterExpr::And(lhs, rhs) => Ok(CompiledFilter::And(
Box::new(self.compile_filter(lhs, now)?),
Box::new(self.compile_filter(rhs, now)?),
)),
FilterExpr::Or(lhs, rhs) => Ok(CompiledFilter::Or(
Box::new(self.compile_filter(lhs, now)?),
Box::new(self.compile_filter(rhs, now)?),
)),
}
}
fn compile_metric_comparison(
&self,
metric: &str,
param: Option<&str>,
op: CompareOp,
value: &FilterValue,
now: Timestamp,
) -> Result<CompiledFilter, String> {
let param = param.ok_or_else(|| {
format!(
"metric comparison on '{metric}' requires a [param] (e.g. '{metric}[some_param]{}...'); \
v1 buckets only carry per-label occurrence data, not a single metric value",
match op {
CompareOp::Lt => "<",
CompareOp::Gt => ">",
CompareOp::Le => "<=",
CompareOp::Ge => ">=",
CompareOp::Eq => "==",
CompareOp::Ne => "!=",
}
)
})?;
if let Some(DataValue::MetricDeclaration { label_params, .. }) =
self.index.get(&IndexKey::MetricDeclaration { name: metric.to_string() })
{
if !label_params.iter().any(|p| p == param) {
return Err(format!(
"param '{param}' is not a label param of metric '{metric}'; declared label params: [{}]",
label_params.join(", ")
));
}
}
let mut matching = HashSet::new();
for (labels, (sequence, _, _)) in self.bucket_values(metric, None) {
let Some(primitive) = labels.get(param) else {
continue;
};
if primitive.compare(op, value, now)? {
matching.insert(*sequence);
}
}
Ok(CompiledFilter::MetricMatch(matching))
}
}
impl CompiledFilter {
fn from_seq_comparison(op: CompareOp, value: &FilterValue) -> Result<Self, String> {
let target = match value {
FilterValue::Number(n) => *n,
FilterValue::Raw(raw) => raw.parse::<f64>().map_err(|_| {
format!("'ottyl_seq' comparisons require a numeric value, got {raw:?}")
})?,
};
Ok(CompiledFilter::Sequence { op, target })
}
fn from_ts_comparison(op: CompareOp, value: &FilterValue, now: Timestamp) -> Result<Self, String> {
let target_ms = match value {
FilterValue::Number(n) => *n as i64,
FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, now)
.map_err(|err| err.to_string())?
.as_millis(),
};
Ok(CompiledFilter::Timestamp { op, target_ms })
}
fn from_fmt_comparison(op: CompareOp, value: &FilterValue) -> Self {
let target = match value {
FilterValue::Number(n) => n.to_string(),
FilterValue::Raw(raw) => raw.clone(),
};
CompiledFilter::Fmt { op, target }
}
}
impl MetricPrimitive {
fn compare(&self, op: CompareOp, value: &FilterValue, now: Timestamp) -> Result<bool, String> {
let ordering = match (self, value) {
(MetricPrimitive::Number(n), FilterValue::Number(v)) => n.as_f64().partial_cmp(v),
(MetricPrimitive::Number(n), FilterValue::Raw(raw)) => {
let v: f64 = raw
.parse()
.map_err(|_| format!("expected a numeric value, got {raw:?}"))?;
n.as_f64().partial_cmp(&v)
}
(MetricPrimitive::Timestamp(ts), FilterValue::Number(v)) => {
ts.as_millis().partial_cmp(&(*v as i64))
}
(MetricPrimitive::Timestamp(ts), FilterValue::Raw(raw)) => {
let target = filter_datetime::parse_datetime(raw, now).map_err(|err| err.to_string())?;
ts.as_millis().partial_cmp(&target.as_millis())
}
(MetricPrimitive::String(s), FilterValue::Raw(raw)) => s.as_str().partial_cmp(raw.as_str()),
(MetricPrimitive::String(s), FilterValue::Number(n)) => {
s.as_str().partial_cmp(n.to_string().as_str())
}
(MetricPrimitive::Duration(ms), FilterValue::Number(v)) => (*ms as f64).partial_cmp(v),
(MetricPrimitive::Duration(ms), FilterValue::Raw(raw)) => {
let v: f64 = raw
.parse()
.map_err(|_| format!("expected a numeric value, got {raw:?}"))?;
(*ms as f64).partial_cmp(&v)
}
(MetricPrimitive::Boolean(b), FilterValue::Raw(raw)) => {
let v: bool = raw
.parse()
.map_err(|_| format!("expected true or false, got {raw:?}"))?;
b.partial_cmp(&v)
}
(MetricPrimitive::Boolean(_), FilterValue::Number(_)) => None,
};
Ok(ordering.is_some_and(|ordering| op.matches(ordering)))
}
}
pub(crate) fn eval_json_comparison(text: &str, field: &str, op: CompareOp, value: &FilterValue) -> bool {
let Ok(parsed) = serde_json::from_str::<serde_json::Value>(text) else {
return false;
};
let Some(field_value) = parsed.as_object().and_then(|obj| obj.get(field)) else {
return false;
};
compare_json_value(field_value, op, value)
}
fn compare_json_value(field_value: &serde_json::Value, op: CompareOp, value: &FilterValue) -> bool {
match field_value {
serde_json::Value::Number(n) => {
let target = match value {
FilterValue::Number(v) => Some(*v),
FilterValue::Raw(raw) => raw.parse::<f64>().ok(),
};
match (n.as_f64(), target) {
(Some(f), Some(v)) => f.partial_cmp(&v).is_some_and(|ordering| op.matches(ordering)),
_ => false,
}
}
serde_json::Value::String(s) => {
let target = match value {
FilterValue::Raw(raw) => raw.clone(),
FilterValue::Number(n) => n.to_string(),
};
s.as_str()
.partial_cmp(target.as_str())
.is_some_and(|ordering| op.matches(ordering))
}
serde_json::Value::Bool(b) => {
let FilterValue::Raw(raw) = value else {
return false;
};
match raw.parse::<bool>() {
Ok(v) => match op {
CompareOp::Eq => *b == v,
CompareOp::Ne => *b != v,
_ => false,
},
Err(_) => false,
}
}
serde_json::Value::Null | serde_json::Value::Array(_) | serde_json::Value::Object(_) => false,
}
}
impl CompareOp {
pub(crate) fn matches(&self, ordering: Ordering) -> bool {
match self {
CompareOp::Lt => ordering == Ordering::Less,
CompareOp::Gt => ordering == Ordering::Greater,
CompareOp::Le => ordering != Ordering::Greater,
CompareOp::Ge => ordering != Ordering::Less,
CompareOp::Eq => ordering == Ordering::Equal,
CompareOp::Ne => ordering != Ordering::Equal,
}
}
}
impl CompiledFilter {
pub(super) fn eval(&self, sequence: u64, timestamp: Timestamp, text: &str) -> bool {
match self {
CompiledFilter::Regex(re) => re.is_match(text),
CompiledFilter::Sequence { op, target } => {
op.matches((sequence as f64).total_cmp(target))
}
CompiledFilter::Timestamp { op, target_ms } => {
op.matches(timestamp.as_millis().cmp(target_ms))
}
CompiledFilter::Fmt { op, target } => {
op.matches(crate::engine::logfmt::detect_format(text).cmp(target.as_str()))
}
CompiledFilter::MetricMatch(sequences) => sequences.contains(&sequence),
CompiledFilter::Json { field, op, value } => eval_json_comparison(text, field, *op, value),
CompiledFilter::And(lhs, rhs) => {
lhs.eval(sequence, timestamp, text) && rhs.eval(sequence, timestamp, text)
}
CompiledFilter::Or(lhs, rhs) => {
lhs.eval(sequence, timestamp, text) || rhs.eval(sequence, timestamp, text)
}
}
}
}
src/engine/snapshot/metrics.rs
+333
View File
@@ -0,0 +1,333 @@
//! Metric declaration and application: validating/storing declarations, applying them to
//! ingested lines to populate metric buckets, and backfilling a newly (re)declared metric
//! against already-ingested logs.
use crate::engine::data_value::{DataValue, MetricPrimitiveKind};
use crate::engine::index_key::IndexKey;
use crate::engine::metric_primitive::{MetricPrimitive, Number};
use crate::engine::snapshot::params::ResolvedParam;
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::timestamp::Timestamp;
use crate::engine::DeclareMetricRequest;
use regex::Regex;
use std::collections::{BTreeMap, HashMap};
impl EngineSnapshot {
pub(crate) fn validate_metric_request(&self, request: &DeclareMetricRequest) -> Result<Regex, String> {
let re = Regex::new(&request.message_match_regex)
.map_err(|err| format!("invalid regex: {err}"))?;
let declared =
|name: &str| self.index.contains_key(&IndexKey::ParamDeclaration { name: name.to_string() });
let mut errors: Vec<String> = request
.label_params
.iter()
.filter(|name| !declared(name))
.map(|name| format!("label param '{name}' references an undeclared param"))
.collect();
if let Some(value_param) = &request.value_param {
if !declared(value_param) {
errors.push(format!("value param '{value_param}' references an undeclared param"));
}
}
if !errors.is_empty() {
return Err(errors.join("; "));
}
Ok(re)
}
pub(crate) fn declare_metric(&mut self, request: DeclareMetricRequest) -> Result<(), String> {
let re = self.validate_metric_request(&request)?;
self.metric_regex_cache.insert(request.name.clone(), re.clone());
let resolved_params = self.resolved_param_declarations();
self.backfill_metric(
&request.name,
&re,
&request.label_params,
request.value_param.as_deref(),
&resolved_params,
request.matched_labels.as_ref(),
request.unmatched_labels.as_ref(),
);
self.index.insert(
IndexKey::MetricDeclaration { name: request.name },
DataValue::MetricDeclaration {
message_match_regex: request.message_match_regex,
label_params: request.label_params,
value_param: request.value_param,
matched_labels: request.matched_labels,
unmatched_labels: request.unmatched_labels,
},
);
Ok(())
}
pub(crate) fn validate_update_metric(
&self,
existing_name: &str,
request: &DeclareMetricRequest,
) -> Result<Regex, String> {
if !self.index.contains_key(&IndexKey::MetricDeclaration { name: existing_name.to_string() }) {
return Err(format!("no metric declaration named '{existing_name}'"));
}
if request.name != existing_name
&& self.index.contains_key(&IndexKey::MetricDeclaration { name: request.name.clone() })
{
return Err(format!("a metric named '{}' already exists", request.name));
}
self.validate_metric_request(request)
}
pub(crate) fn update_metric(&mut self, existing_name: &str, request: DeclareMetricRequest) -> Result<(), String> {
let re = self.validate_update_metric(existing_name, &request)?;
let resolved_params = self.resolved_param_declarations();
self.clear_metric_data(existing_name);
if request.name != existing_name {
self.index.remove(&IndexKey::MetricDeclaration { name: existing_name.to_string() });
self.metric_regex_cache.remove(existing_name);
}
self.metric_regex_cache.insert(request.name.clone(), re.clone());
self.backfill_metric(
&request.name,
&re,
&request.label_params,
request.value_param.as_deref(),
&resolved_params,
request.matched_labels.as_ref(),
request.unmatched_labels.as_ref(),
);
self.index.insert(
IndexKey::MetricDeclaration { name: request.name.clone() },
DataValue::MetricDeclaration {
message_match_regex: request.message_match_regex,
label_params: request.label_params,
value_param: request.value_param,
matched_labels: request.matched_labels,
unmatched_labels: request.unmatched_labels,
},
);
Ok(())
}
fn clear_metric_data(&mut self, metric_name: &str) {
self.index.retain(|key, _| match key {
IndexKey::Bucket { metric_name: name, .. } => name != metric_name,
IndexKey::LastMetricEvent { metric_name: name, .. } => name != metric_name,
_ => true,
});
}
pub(crate) fn remove_metric_declaration(&mut self, name: &str) -> bool {
let existed = self.index.remove(&IndexKey::MetricDeclaration { name: name.to_string() }).is_some();
self.metric_regex_cache.remove(name);
self.clear_metric_data(name);
existed
}
}
pub(super) fn parse_metric_primitive(raw: &str, kind: MetricPrimitiveKind) -> Option<MetricPrimitive> {
match kind {
MetricPrimitiveKind::String => Some(MetricPrimitive::String(raw.to_string())),
MetricPrimitiveKind::Number => raw
.parse::<i64>()
.map(Number::Int)
.or_else(|_| raw.parse::<f64>().map(Number::Float))
.ok()
.map(MetricPrimitive::Number),
MetricPrimitiveKind::Timestamp => {
crate::engine::filter_datetime::parse_datetime(raw, Timestamp::now())
.ok()
.map(MetricPrimitive::Timestamp)
}
MetricPrimitiveKind::Duration => raw.parse::<i64>().ok().map(MetricPrimitive::Duration),
MetricPrimitiveKind::Boolean => raw.parse::<bool>().ok().map(MetricPrimitive::Boolean),
}
}
impl EngineSnapshot {
#[allow(clippy::too_many_arguments)]
fn apply_metric_to_log(
&mut self,
metric_name: &str,
regex: &Regex,
label_params: &[String],
value_param: Option<&str>,
resolved_params: &HashMap<String, ResolvedParam>,
matched_labels: Option<&BTreeMap<String, String>>,
unmatched_labels: Option<&BTreeMap<String, String>>,
sequence: u64,
timestamp: Timestamp,
text: &str,
) {
if regex.captures(text).is_none() {
if let Some(unmatched_labels) = unmatched_labels {
let labels = unmatched_labels
.iter()
.map(|(name, value)| (name.clone(), MetricPrimitive::String(value.clone())))
.collect();
self.push_bucket_value(metric_name, labels, sequence, timestamp, MetricPrimitive::Number(Number::Int(1)));
}
return;
}
let mut labels = BTreeMap::new();
for param_name in label_params {
let Some(resolved) = resolved_params.get(param_name) else { return };
let Some(parsed) = self.resolve_param_value(metric_name, resolved, &labels, text, timestamp) else {
return;
};
labels.insert(param_name.clone(), parsed);
}
if let Some(matched_labels) = matched_labels {
for (name, value) in matched_labels {
labels.insert(name.clone(), MetricPrimitive::String(value.clone()));
}
}
let value = match value_param {
Some(value_param) => {
let Some(resolved) = resolved_params.get(value_param) else { return };
match self.resolve_param_value(metric_name, resolved, &labels, text, timestamp) {
Some(parsed) => parsed,
None => return,
}
}
None => MetricPrimitive::Number(Number::Int(1)),
};
self.push_bucket_value(metric_name, labels, sequence, timestamp, value);
}
pub(crate) fn push_bucket_value(
&mut self,
metric_name: &str,
labels: BTreeMap<String, MetricPrimitive>,
sequence: u64,
timestamp: Timestamp,
value: MetricPrimitive,
) {
let key = IndexKey::Bucket { metric_name: metric_name.to_string(), labels };
let entry = self
.index
.entry(key)
.or_insert_with(|| DataValue::BucketValue { values: Vec::new() });
if let DataValue::BucketValue { values } = entry {
values.push((sequence, timestamp, value));
}
}
#[allow(clippy::too_many_arguments)]
fn backfill_metric(
&mut self,
metric_name: &str,
regex: &Regex,
label_params: &[String],
value_param: Option<&str>,
resolved_params: &HashMap<String, ResolvedParam>,
matched_labels: Option<&BTreeMap<String, String>>,
unmatched_labels: Option<&BTreeMap<String, String>>,
) {
let lines: Vec<(u64, Timestamp, String)> = self
.logs_range()
.map(|(sequence, timestamp, text)| (sequence, timestamp, text.to_string()))
.collect();
for (sequence, timestamp, text) in lines {
self.apply_metric_to_log(
metric_name,
regex,
label_params,
value_param,
resolved_params,
matched_labels,
unmatched_labels,
sequence,
timestamp,
&text,
);
}
}
pub(crate) fn apply_all_metrics(&mut self, sequence: u64, timestamp: Timestamp, text: &str) {
type SeededMetric = (
String,
Vec<String>,
Option<String>,
Option<BTreeMap<String, String>>,
Option<BTreeMap<String, String>>,
);
let metrics: Vec<SeededMetric> = self
.metric_declarations_range()
.map(|(name, _, label_params, value_param, matched_labels, unmatched_labels)| {
(
name.to_string(),
label_params.to_vec(),
value_param.map(|s| s.to_string()),
matched_labels.cloned(),
unmatched_labels.cloned(),
)
})
.collect();
let resolved_params = self.resolved_param_declarations();
for (name, label_params, value_param, matched_labels, unmatched_labels) in metrics {
if let Some(re) = self.metric_regex_cache.get(&name).cloned() {
self.apply_metric_to_log(
&name,
&re,
&label_params,
value_param.as_deref(),
&resolved_params,
matched_labels.as_ref(),
unmatched_labels.as_ref(),
sequence,
timestamp,
text,
);
}
}
}
pub(crate) fn bucket_values<'a>(
&'a self,
metric_name: &'a str,
labels_filter: Option<&'a BTreeMap<String, MetricPrimitive>>,
) -> impl Iterator<Item = (&'a BTreeMap<String, MetricPrimitive>, &'a (u64, Timestamp, MetricPrimitive))> + 'a
{
self.index
.range(IndexKey::MetricBucketSplit..IndexKey::End)
.filter_map(move |(key, value)| match (key, value) {
(IndexKey::Bucket { metric_name: name, labels }, DataValue::BucketValue { values })
if name == metric_name && labels_filter.is_none_or(|filter| filter == labels) =>
{
Some(values.iter().map(move |entry| (labels, entry)))
}
_ => None,
})
.flatten()
}
}
impl EngineSnapshot {
pub(crate) fn declared_value_kind(&self, metric_name: &str) -> Option<MetricPrimitiveKind> {
match self.index.get(&IndexKey::MetricDeclaration { name: metric_name.to_string() }) {
Some(DataValue::MetricDeclaration { value_param, .. }) => Some(
value_param
.as_ref()
.and_then(|name| {
self.index.get(&IndexKey::ParamDeclaration { name: name.clone() })
})
.map(|value| match value {
DataValue::ParamDeclaration(param) => param.kind,
_ => MetricPrimitiveKind::Number,
})
.unwrap_or(MetricPrimitiveKind::Number),
),
_ => None,
}
}
}
src/engine/snapshot/mod.rs
+192
View File
@@ -0,0 +1,192 @@
//! `EngineSnapshot` is the actor's live state: ingested log lines, declared params/metrics,
//! computed metric buckets, and the sort-helper indexes built on demand — all stored in one
//! flat `BTreeMap<IndexKey, DataValue>` and read/written through the left-right index in
//! `super::handle`. Its impl is split across the submodules below by concern; this file holds
//! the struct itself plus the accessors every submodule builds on.
pub(crate) mod filter;
pub(crate) mod metrics;
pub(crate) mod params;
pub(crate) mod query;
pub(crate) mod sort;
pub(crate) use filter::eval_json_comparison;
use crate::engine::data_value::{DataValue, MetricPrimitiveKind, ParamDeclaration};
use crate::engine::index_key::IndexKey;
use crate::engine::message::{EngineQuery, EngineResponse};
use crate::engine::timestamp::Timestamp;
use query::handle_query_inner;
use regex::Regex;
use sort::{SortHelperAccess, SortHelperIndexEntry};
use std::collections::{BTreeMap, HashMap};
#[derive(Debug, Clone, Default)]
pub(crate) struct EngineSnapshot {
pub(crate) index: BTreeMap<IndexKey, DataValue>,
pub(crate) sort_helper_indexes: HashMap<String, SortHelperIndexEntry>,
/// Compiled regexes for param/metric declarations, keyed by declaration name. Compiling a
/// regex is far more expensive than matching with one, and `apply_all_metrics`/
/// `update_sort_helper_indexes_on_ingest` run per ingested line, so every insertion or
/// removal of a declaration (see `params.rs`/`metrics.rs`) must keep these in sync rather
/// than recompiling from the stored pattern string on every line.
pub(crate) param_regex_cache: HashMap<String, Regex>,
pub(crate) metric_regex_cache: HashMap<String, Regex>,
pub(crate) log_count: usize,
pub(crate) approx_bytes_total: u64,
pub(crate) ingest_lag_high_watermark_ms: i64,
}
impl EngineSnapshot {
pub(crate) fn metric_declarations_range(
&self,
) -> impl Iterator<
Item = (
&str,
&str,
&[String],
Option<&str>,
Option<&BTreeMap<String, String>>,
Option<&BTreeMap<String, String>>,
),
> {
self.index
.range(IndexKey::ParamMetricSplit..IndexKey::MetricBucketSplit)
.filter_map(|(key, value)| match (key, value) {
(
IndexKey::MetricDeclaration { name },
DataValue::MetricDeclaration {
message_match_regex,
label_params,
value_param,
matched_labels,
unmatched_labels,
},
) => Some((
name.as_str(),
message_match_regex.as_str(),
label_params.as_slice(),
value_param.as_deref(),
matched_labels.as_ref(),
unmatched_labels.as_ref(),
)),
_ => None,
})
}
pub(crate) fn param_declarations_range(
&self,
) -> impl Iterator<Item = (&str, &str, &str, Option<&str>, MetricPrimitiveKind, bool, bool, bool)> {
self.index
.range(IndexKey::LogParamSplit..IndexKey::ParamMetricSplit)
.filter_map(|(key, value)| match (key, value) {
(
IndexKey::ParamDeclaration { name },
DataValue::ParamDeclaration(ParamDeclaration {
name: _,
message_match_regex,
capture_group,
json_field_path,
kind,
elapsed_since_last,
auto_derived,
log_stream_gated,
}),
) => Some((
name.as_str(),
message_match_regex.as_str(),
capture_group.as_str(),
json_field_path.as_deref(),
*kind,
*elapsed_since_last,
*auto_derived,
*log_stream_gated,
)),
_ => None,
})
}
}
pub(crate) fn approx_log_bytes(text: &str) -> u64 {
text.len() as u64
}
impl EngineSnapshot {
pub(crate) fn from_index(index: BTreeMap<IndexKey, DataValue>) -> Self {
let mut snapshot = EngineSnapshot { index, ..Default::default() };
snapshot.rebuild_regex_caches();
snapshot
}
/// Recompiles `param_regex_cache`/`metric_regex_cache` from the declarations currently in
/// `index`. Needed whenever a snapshot's declarations are populated by inserting straight
/// into `index` rather than through the normal declare/insert methods that keep the caches
/// in sync incrementally (e.g. `from_index` building a snapshot from a bare index).
pub(crate) fn rebuild_regex_caches(&mut self) {
self.param_regex_cache = self
.param_declarations_range()
.filter_map(|(name, message_match_regex, ..)| {
regex::Regex::new(message_match_regex).ok().map(|re| (name.to_string(), re))
})
.collect();
self.metric_regex_cache = self
.metric_declarations_range()
.filter_map(|(name, message_match_regex, ..)| {
regex::Regex::new(message_match_regex).ok().map(|re| (name.to_string(), re))
})
.collect();
}
pub(crate) fn evict_excess_logs(&mut self, max_logs: Option<usize>) {
let Some(max_logs) = max_logs else { return };
while self.log_count > max_logs {
let Some(oldest_key) = self
.index
.range(IndexKey::Start..IndexKey::LogParamSplit)
.next()
.map(|(key, _)| key.clone())
else {
break;
};
if let Some(DataValue::LogLiteral { text }) = self.index.get(&oldest_key) {
self.approx_bytes_total = self.approx_bytes_total.saturating_sub(approx_log_bytes(text));
}
self.index.remove(&oldest_key);
self.log_count -= 1;
}
}
pub(crate) fn logs_range(&self) -> impl Iterator<Item = (u64, Timestamp, &str)> {
self.index.range(IndexKey::Start..IndexKey::LogParamSplit).filter_map(|(key, value)| {
match (key, value) {
(IndexKey::IngestedLog { sequence, timestamp }, DataValue::LogLiteral { text }) => {
Some((*sequence, *timestamp, text.as_str()))
}
_ => None,
}
})
}
pub(crate) fn logs_since(
&self,
before_sequence: u64,
since_ms: i64,
now_ms: i64,
) -> impl Iterator<Item = (u64, Timestamp, &str)> {
self.logs_range().filter(move |(sequence, timestamp, _)| {
*sequence < before_sequence
&& timestamp.as_millis() > since_ms
&& timestamp.as_millis() <= now_ms
})
}
}
impl EngineSnapshot {
pub(crate) fn handle_query_ro(&self, query: EngineQuery) -> EngineResponse {
handle_query_inner(self, query, SortHelperAccess::Immutable(&self.sort_helper_indexes))
}
pub(crate) fn has_sort_helper_index(&self, sort_by: &str) -> bool {
self.sort_helper_indexes.contains_key(sort_by)
}
}
src/engine/snapshot/params.rs
+360
View File
@@ -0,0 +1,360 @@
//! Param declaration: validating and storing explicit declarations, and the auto-detection
//! logic that derives declarations from logfmt/json/log-stream-shaped lines as they're ingested.
use crate::consts::{MAX_AUTO_DECLARED_PARAMS, MAX_NEW_AUTO_PARAMS_PER_LINE, OTTYL_RESERVED_PARAM_NAMES};
use crate::engine::data_value::{DataValue, MetricPrimitiveKind, ParamDeclaration};
use crate::engine::index_key::IndexKey;
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::snapshot::metrics::parse_metric_primitive;
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::timestamp::Timestamp;
use crate::engine::DeclareParamRequest;
use regex::Regex;
use std::collections::{BTreeMap, HashMap};
impl EngineSnapshot {
pub(crate) fn validate_param_declaration(&self, request: &DeclareParamRequest) -> Result<(), String> {
if OTTYL_RESERVED_PARAM_NAMES.contains(&request.name.as_str()) {
return Err(format!(
"'{}' is a reserved name, present on every log line without declaring anything; \
it can't be redeclared as a param",
request.name
));
}
if let Some(DataValue::ParamDeclaration(existing)) =
self.index.get(&IndexKey::ParamDeclaration { name: request.name.clone() })
{
if existing.auto_derived {
return Err(format!(
"param '{}' was auto-detected from the log stream and can't be edited",
request.name
));
}
}
let re = Regex::new(&request.message_match_regex)
.map_err(|err| format!("invalid regex: {err}"))?;
if request.elapsed_since_last && request.kind != MetricPrimitiveKind::Duration {
return Err("elapsed_since_last params must declare kind 'duration'".to_string());
}
let uses_json_field = request.json_field_path.as_deref().is_some_and(|path| !path.is_empty());
if request.kind != MetricPrimitiveKind::Boolean
&& !request.elapsed_since_last
&& !uses_json_field
&& !re.capture_names().flatten().any(|name| name == request.capture_group)
{
return Err(format!(
"regex has no capture group named '{}'",
request.capture_group
));
}
Ok(())
}
pub(crate) fn insert_param_declaration(&mut self, request: DeclareParamRequest) {
if let Ok(re) = Regex::new(&request.message_match_regex) {
self.param_regex_cache.insert(request.name.clone(), re);
}
self.index.insert(
IndexKey::ParamDeclaration { name: request.name.clone() },
DataValue::ParamDeclaration(ParamDeclaration { auto_derived: false, ..request }),
);
}
}
#[cfg(test)]
pub(crate) fn declare_param(snapshot: &mut EngineSnapshot, request: DeclareParamRequest) -> Result<(), String> {
snapshot.validate_param_declaration(&request)?;
snapshot.insert_param_declaration(request);
Ok(())
}
fn sanitize_group_name(key: &str) -> String {
let mut group: String =
key.chars().map(|c| if c.is_ascii_alphanumeric() || c == '_' { c } else { '_' }).collect();
if group.chars().next().is_none_or(|c| c.is_ascii_digit()) {
group.insert(0, '_');
}
group
}
fn infer_kind(sample: &str) -> MetricPrimitiveKind {
if sample.parse::<i64>().is_ok() || sample.parse::<f64>().is_ok() {
MetricPrimitiveKind::Number
} else {
MetricPrimitiveKind::String
}
}
fn build_auto_param_pattern(key: &str, group: &str, quoted: bool) -> String {
let escaped_key = regex::escape(key);
if quoted {
format!(r#"{escaped_key}="(?P<{group}>[^"]*)""#)
} else {
format!(r"{escaped_key}=(?P<{group}>\S+)")
}
}
struct AutoParamCandidate {
name: String,
value: String,
message_match_regex: String,
capture_group: String,
json_field_path: Option<String>,
}
impl EngineSnapshot {
fn auto_derived_param_count(&self) -> usize {
self.param_declarations_range().filter(|declaration| declaration.6).count()
}
pub(crate) fn auto_declare_params(&mut self, text: &str) {
let mut total = self.auto_derived_param_count();
if total >= MAX_AUTO_DECLARED_PARAMS {
return;
}
let candidates: Vec<AutoParamCandidate> = if crate::engine::logfmt::detect_format(text) == "json" {
crate::engine::data_value::flatten_json_object(text)
.into_iter()
.map(|(path, value)| AutoParamCandidate {
name: path.clone(),
value,
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: Some(path),
})
.collect()
} else {
crate::engine::logfmt::extract_pairs(text)
.into_iter()
.map(|(key, value, quoted)| {
let group = sanitize_group_name(key);
AutoParamCandidate {
name: key.to_string(),
value: value.to_string(),
message_match_regex: build_auto_param_pattern(key, &group, quoted),
capture_group: group,
json_field_path: None,
}
})
.collect()
};
let mut new_this_line = 0;
for candidate in candidates {
if new_this_line >= MAX_NEW_AUTO_PARAMS_PER_LINE || total >= MAX_AUTO_DECLARED_PARAMS {
break;
}
let index_key = IndexKey::ParamDeclaration { name: candidate.name.clone() };
if self.index.contains_key(&index_key) {
continue;
}
if let Ok(re) = Regex::new(&candidate.message_match_regex) {
self.param_regex_cache.insert(candidate.name.clone(), re);
}
self.index.insert(
index_key,
DataValue::ParamDeclaration(ParamDeclaration {
name: candidate.name,
message_match_regex: candidate.message_match_regex,
capture_group: candidate.capture_group,
json_field_path: candidate.json_field_path,
kind: infer_kind(&candidate.value),
elapsed_since_last: false,
auto_derived: true,
log_stream_gated: false,
}),
);
new_this_line += 1;
total += 1;
}
}
}
pub(crate) fn log_stream_header_transition(
current: &Option<crate::engine::logfmt::LogStreamHeader>,
text: &str,
) -> Option<crate::engine::logfmt::LogStreamHeader> {
crate::engine::logfmt::LogStreamHeader::detect(text).or_else(|| current.clone())
}
impl EngineSnapshot {
pub(crate) fn derive_log_stream_params_from_row(
&mut self,
header: &crate::engine::logfmt::LogStreamHeader,
text: &str,
) {
if !crate::engine::logfmt::is_log_stream_line(text) {
return;
}
let row = header.split_row(text);
for (i, column) in header.columns.iter().enumerate() {
let Some(value) = row.get(i) else { continue };
let index_key = IndexKey::ParamDeclaration { name: column.name.clone() };
if self.index.contains_key(&index_key) {
continue;
}
let message_match_regex = header.column_pattern(i, &column.name);
if let Ok(re) = Regex::new(&message_match_regex) {
self.param_regex_cache.insert(column.name.clone(), re);
}
self.index.insert(
index_key,
DataValue::ParamDeclaration(ParamDeclaration {
name: column.name.clone(),
message_match_regex,
capture_group: column.name.clone(),
json_field_path: None,
kind: infer_kind(value.trim()),
elapsed_since_last: false,
auto_derived: true,
log_stream_gated: true,
}),
);
}
if let Some(message) = row.get(header.columns.len()).filter(|m| !m.trim().is_empty()) {
let index_key = IndexKey::ParamDeclaration { name: "message".to_string() };
if !self.index.contains_key(&index_key) {
let message_match_regex = header.message_pattern("message");
if let Ok(re) = Regex::new(&message_match_regex) {
self.param_regex_cache.insert("message".to_string(), re);
}
self.index.insert(
index_key,
DataValue::ParamDeclaration(ParamDeclaration {
name: "message".to_string(),
message_match_regex,
capture_group: "message".to_string(),
json_field_path: None,
kind: infer_kind(message.trim()),
elapsed_since_last: false,
auto_derived: true,
log_stream_gated: true,
}),
);
}
}
}
}
#[cfg(test)]
pub(crate) fn auto_declare_log_stream_params(
snapshot: &mut EngineSnapshot,
header: &mut Option<crate::engine::logfmt::LogStreamHeader>,
text: &str,
) {
let is_new_header = crate::engine::logfmt::LogStreamHeader::detect(text).is_some();
*header = log_stream_header_transition(header, text);
if is_new_header {
return;
}
if let Some(header) = header.clone() {
snapshot.derive_log_stream_params_from_row(&header, text);
}
}
impl EngineSnapshot {
pub(crate) fn clear_all_param_declarations(&mut self) -> usize {
let names: Vec<String> =
self.param_declarations_range().map(|(name, ..)| name.to_string()).collect();
let removed = names.len();
for name in names {
self.index.remove(&IndexKey::ParamDeclaration { name: name.clone() });
self.param_regex_cache.remove(&name);
}
removed
}
pub(crate) fn reextract_params_from_retained_logs(&mut self) -> (usize, usize) {
let texts: Vec<String> = self.logs_range().map(|(_, _, text)| text.to_string()).collect();
let logs_considered = texts.len();
for text in &texts {
self.auto_declare_params(text);
}
let reextracted = self.param_declarations_range().count();
(reextracted, logs_considered)
}
}
pub(super) struct ResolvedParam {
regex: Regex,
capture_group: String,
json_field_path: Option<String>,
kind: MetricPrimitiveKind,
elapsed_since_last: bool,
log_stream_gated: bool,
}
impl EngineSnapshot {
pub(super) fn resolved_param_declarations(&self) -> HashMap<String, ResolvedParam> {
self.param_declarations_range()
.filter_map(|(name, _, capture_group, json_field_path, kind, elapsed_since_last, _, log_stream_gated)| {
self.param_regex_cache.get(name).map(|regex| {
(
name.to_string(),
ResolvedParam {
regex: regex.clone(),
capture_group: capture_group.to_string(),
json_field_path: json_field_path.map(|path| path.to_string()),
kind,
elapsed_since_last,
log_stream_gated,
},
)
})
})
.collect()
}
}
fn extract_param_raw(
text: &str,
regex: &Regex,
capture_group: &str,
json_field_path: Option<&str>,
log_stream_gated: bool,
) -> Option<String> {
if log_stream_gated && !crate::engine::logfmt::is_log_stream_line(text) {
return None;
}
match json_field_path.filter(|path| !path.is_empty()) {
Some(field_path) => crate::engine::data_value::extract_json_field(text, field_path),
None => regex.captures(text)?.name(capture_group).map(|m| m.as_str().to_string()),
}
}
impl EngineSnapshot {
pub(super) fn resolve_param_value(
&mut self,
metric_name: &str,
resolved: &ResolvedParam,
labels_so_far: &BTreeMap<String, MetricPrimitive>,
text: &str,
timestamp: Timestamp,
) -> Option<MetricPrimitive> {
if resolved.elapsed_since_last {
let last_event_key = IndexKey::LastMetricEvent {
metric_name: metric_name.to_string(),
labels: labels_so_far.clone(),
};
let previous_timestamp = match self.index.get(&last_event_key) {
Some(DataValue::LastEventTimestamp { timestamp: previous }) => Some(*previous),
_ => None,
};
self.index.insert(last_event_key, DataValue::LastEventTimestamp { timestamp });
return previous_timestamp
.map(|previous| MetricPrimitive::Duration(timestamp.as_millis() - previous.as_millis()));
}
let raw = extract_param_raw(
text,
&resolved.regex,
&resolved.capture_group,
resolved.json_field_path.as_deref(),
resolved.log_stream_gated,
)?;
parse_metric_primitive(&raw, resolved.kind)
}
}
src/engine/snapshot/query.rs
+549
View File
@@ -0,0 +1,549 @@
//! Answering `EngineQuery`s: log/timeline listing (including the sort-helper-index path),
//! stats, declaration listing, bucket-value listing, and PromQL evaluation.
use crate::api::dto::{
BucketValueDto, LogEntryDto, MetricDeclarationDto, ParamDeclarationDto, PromqlSeriesDto,
StatsDto, TimelineBucketDto,
};
use crate::consts::{DEFAULT_TIMELINE_BUCKET_MS, MERGE_WINDOW_MS};
use crate::engine::data_value::{DataValue, MetricPrimitiveKind};
use crate::engine::index_key::IndexKey;
use crate::engine::message::{EngineQuery, EngineResponse};
use crate::engine::metric_primitive::{MetricPrimitive, Number};
use crate::engine::promql::{AggOp, LabelMatcher, PromqlQuery};
use crate::engine::rate;
#[cfg(test)]
use crate::engine::snapshot::sort::SortHelperIndexEntry;
use crate::engine::snapshot::sort::{log_entry_params, SortHelperAccess, SortHelperKey};
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::timestamp::Timestamp;
#[cfg(test)]
use std::collections::HashMap;
use std::collections::BTreeMap;
#[cfg(test)]
pub(crate) fn handle_query(
snapshot: &EngineSnapshot,
query: EngineQuery,
_approx_bytes_total: u64,
_ingest_lag_high_watermark_ms: i64,
sort_helper_indexes: &mut HashMap<String, SortHelperIndexEntry>,
) -> EngineResponse {
handle_query_inner(snapshot, query, SortHelperAccess::Mutable(sort_helper_indexes))
}
pub(super) fn handle_query_inner(
snapshot: &EngineSnapshot,
query: EngineQuery,
sort_helper_access: SortHelperAccess<'_>,
) -> EngineResponse {
let index = &snapshot.index;
let approx_bytes_total = snapshot.approx_bytes_total;
let ingest_lag_high_watermark_ms = snapshot.ingest_lag_high_watermark_ms;
match query {
EngineQuery::Logs(params) => {
let compiled_filter = match params.filter.as_deref() {
Some(filter) => match snapshot.compile_filter_query(filter, Timestamp::now()) {
Ok(compiled) => Some(compiled),
Err(err) => return EngineResponse::Error(err),
},
None => None,
};
if let Some(sort_by) = params.sort_by.as_deref() {
if !snapshot.param_declarations_range().any(|(name, ..)| name == sort_by) {
return EngineResponse::Error(format!("unknown param '{sort_by}' for sort_by"));
}
}
let limit = params.limit.unwrap_or(500).min(10_000);
let compiled_params = snapshot.compiled_param_declarations();
let filtered = snapshot
.logs_range()
.filter(|(sequence, timestamp, _)| {
params.since_sequence.is_none_or(|since| *sequence > since)
&& params.start_ms.is_none_or(|start| timestamp.as_millis() >= start)
&& params.end_ms.is_none_or(|end| timestamp.as_millis() <= end)
})
.filter(|(sequence, timestamp, text)| {
compiled_filter
.as_ref()
.is_none_or(|filter| filter.eval(*sequence, *timestamp, text))
});
let entries: Vec<LogEntryDto> = match params.sort_by.as_deref() {
None => filtered
.take(limit)
.map(|(sequence, timestamp, text)| LogEntryDto {
sequence,
timestamp_ms: timestamp.as_millis(),
text: text.to_string(),
params: log_entry_params(text, &compiled_params),
})
.collect(),
Some(sort_by) => {
static EMPTY_HELPER: std::sync::LazyLock<BTreeMap<SortHelperKey, Timestamp>> =
std::sync::LazyLock::new(BTreeMap::new);
let helper_index =
sort_helper_access.resolve(index, sort_by).unwrap_or(&EMPTY_HELPER);
let sort_missing_fallback = params.sort_missing_fallback.unwrap_or(true);
let mut entries = Vec::new();
for (helper_key, timestamp) in helper_index.iter() {
let sequence = match helper_key {
SortHelperKey::WithParam(_, sequence) => *sequence,
SortHelperKey::WithoutParam(sequence) => {
if !sort_missing_fallback {
continue;
}
*sequence
}
};
if !(params.since_sequence.is_none_or(|since| sequence > since)
&& params.start_ms.is_none_or(|start| timestamp.as_millis() >= start)
&& params.end_ms.is_none_or(|end| timestamp.as_millis() <= end))
{
continue;
}
let Some(DataValue::LogLiteral { text }) =
index.get(&IndexKey::IngestedLog { sequence, timestamp: *timestamp })
else {
continue;
};
if !compiled_filter
.as_ref()
.is_none_or(|filter| filter.eval(sequence, *timestamp, text))
{
continue;
}
entries.push(LogEntryDto {
sequence,
timestamp_ms: timestamp.as_millis(),
text: text.clone(),
params: log_entry_params(text, &compiled_params),
});
if entries.len() >= limit {
break;
}
}
entries
}
};
let entries: Vec<LogEntryDto> = entries.into_iter().take(limit).collect();
EngineResponse::Logs(entries)
}
EngineQuery::Timeline(params) => {
let compiled_filter = match params.filter.as_deref() {
Some(filter) => match snapshot.compile_filter_query(filter, Timestamp::now()) {
Ok(compiled) => Some(compiled),
Err(err) => return EngineResponse::Error(err),
},
None => None,
};
let bucket_ms = params.bucket_ms.unwrap_or(DEFAULT_TIMELINE_BUCKET_MS).max(1);
let mut buckets: BTreeMap<i64, u64> = BTreeMap::new();
for (sequence, timestamp, text) in snapshot.logs_range().filter(|(_, timestamp, _)| {
params.start_ms.is_none_or(|start| timestamp.as_millis() >= start)
&& params.end_ms.is_none_or(|end| timestamp.as_millis() <= end)
}) {
if compiled_filter
.as_ref()
.is_some_and(|filter| !filter.eval(sequence, timestamp, text))
{
continue;
}
let bucket_start = (timestamp.as_millis() / bucket_ms) * bucket_ms;
*buckets.entry(bucket_start).or_insert(0) += 1;
}
let out = buckets
.into_iter()
.map(|(start_ms, count)| TimelineBucketDto { start_ms, count })
.collect();
EngineResponse::Timeline(out)
}
EngineQuery::Stats => {
let mut log_count = 0u64;
let mut first_sequence = None;
let mut last_sequence = None;
let mut first_timestamp_ms = None;
let mut last_timestamp_ms = None;
for (sequence, timestamp, _) in snapshot.logs_range() {
log_count += 1;
first_sequence.get_or_insert(sequence);
last_sequence = Some(sequence);
first_timestamp_ms.get_or_insert(timestamp.as_millis());
last_timestamp_ms = Some(timestamp.as_millis());
}
EngineResponse::Stats(StatsDto {
log_count,
first_sequence,
last_sequence,
first_timestamp_ms,
last_timestamp_ms,
approx_bytes_total,
ingest_lag_high_watermark_ms,
})
}
EngineQuery::ParamDeclarations => {
let declarations = snapshot.param_declarations_range()
.map(|(name, message_match_regex, capture_group, json_field_path, kind, elapsed_since_last, auto_derived, _)| {
ParamDeclarationDto {
name: name.to_string(),
message_match_regex: message_match_regex.to_string(),
capture_group: capture_group.to_string(),
json_field_path: json_field_path.map(|path| path.to_string()),
kind,
elapsed_since_last,
auto_derived,
reserved: false,
}
})
.collect();
EngineResponse::ParamDeclarations(declarations)
}
EngineQuery::MetricDeclarations => {
let declarations = snapshot.metric_declarations_range()
.map(|(name, message_match_regex, label_params, value_param, matched_labels, unmatched_labels)| {
MetricDeclarationDto {
name: name.to_string(),
message_match_regex: message_match_regex.to_string(),
label_params: label_params.to_vec(),
value_param: value_param.map(|s| s.to_string()),
matched_labels: matched_labels.cloned(),
unmatched_labels: unmatched_labels.cloned(),
}
})
.collect();
EngineResponse::MetricDeclarations(declarations)
}
EngineQuery::BucketValues {
metric_name,
labels_filter,
start_ms,
end_ms,
since_sequence,
limit,
} => {
let limit = limit.unwrap_or(500).min(10_000);
let values: Vec<BucketValueDto> = snapshot.bucket_values(&metric_name, None)
.filter(|(labels, _)| {
labels_filter.as_ref().is_none_or(|filter| {
filter.iter().all(|(name, expected)| {
labels
.get(name)
.is_some_and(|value| &value.to_string() == expected)
})
})
})
.filter(|(_, (sequence, timestamp, _))| {
since_sequence.is_none_or(|since| *sequence > since)
&& start_ms.is_none_or(|start| timestamp.as_millis() >= start)
&& end_ms.is_none_or(|end| timestamp.as_millis() <= end)
})
.take(limit)
.map(|(labels, (sequence, timestamp, value))| BucketValueDto {
labels: labels.clone(),
sequence: *sequence,
timestamp_ms: timestamp.as_millis(),
value: value.clone(),
})
.collect();
EngineResponse::BucketValues(values)
}
EngineQuery::Promql { query, start_ms, end_ms } => {
let response = PromqlQuery::parse(&query)
.and_then(|parsed| snapshot.evaluate_promql(&parsed, start_ms, end_ms));
EngineResponse::Promql(response)
}
}
}
fn coerce_to_f64(value: &MetricPrimitive) -> f64 {
match value {
MetricPrimitive::Number(n) => match n {
Number::Int(v) => *v as f64,
Number::Float(v) => *v,
},
MetricPrimitive::Timestamp(ts) => ts.as_millis() as f64,
MetricPrimitive::Duration(ms) => *ms as f64,
MetricPrimitive::String(_) => 1.0,
MetricPrimitive::Boolean(b) => {
if *b {
1.0
} else {
0.0
}
}
}
}
fn matches_label_matchers(
labels: &BTreeMap<String, MetricPrimitive>,
matchers: &[LabelMatcher],
) -> bool {
matchers.iter().all(|matcher| {
labels.get(&matcher.name).is_some_and(|value| value.to_string() == matcher.value)
})
}
fn project_labels(
labels: &BTreeMap<String, String>,
by_labels: &[String],
) -> BTreeMap<String, String> {
by_labels
.iter()
.filter_map(|name| labels.get(name).map(|value| (name.clone(), value.clone())))
.collect()
}
fn windowed_rate_series(
points: &[(i64, f64)],
range_ms: i64,
start_ms: Option<i64>,
end_ms: Option<i64>,
) -> Vec<(i64, f64)> {
let ts_points: Vec<(Timestamp, f64)> =
points.iter().map(|(ms, value)| (Timestamp::from_millis(*ms), *value)).collect();
let mut series: Vec<(i64, f64)> = rate::windowed_rate_series(&ts_points, range_ms)
.into_iter()
.map(|(ts, value)| (ts.as_millis(), value))
.collect();
if let Some(end) = end_ms {
if series.last().is_none_or(|(ts, _)| *ts < end) {
let value = rate::rate_at(&ts_points, Timestamp::from_millis(end), range_ms);
series.push((end, value));
}
}
if let Some(start) = start_ms {
if series.first().is_none_or(|(ts, _)| *ts > start) {
let value = rate::rate_at(&ts_points, Timestamp::from_millis(start), range_ms);
series.insert(0, (start, value));
}
}
series
}
fn merge_series_windowed(sub_series: &[Vec<(i64, f64)>], is_avg: bool) -> Vec<(i64, f64)> {
if sub_series.len() <= 1 {
return sub_series.first().cloned().unwrap_or_default();
}
let mut grouped: BTreeMap<i64, Vec<f64>> = BTreeMap::new();
for points in sub_series {
for (timestamp_ms, value) in points {
let window_start = timestamp_ms.div_euclid(MERGE_WINDOW_MS) * MERGE_WINDOW_MS;
grouped.entry(window_start).or_default().push(*value);
}
}
grouped
.into_iter()
.map(|(timestamp_ms, values)| {
let sum: f64 = values.iter().sum();
let value = if is_avg { sum / values.len() as f64 } else { sum };
(timestamp_ms, value)
})
.collect()
}
fn merge_series_locf(sub_series: &[Vec<(i64, f64)>], is_avg: bool) -> Vec<(i64, f64)> {
let mut all_timestamps: std::collections::BTreeSet<i64> = std::collections::BTreeSet::new();
for points in sub_series {
all_timestamps.extend(points.iter().map(|(ts, _)| *ts));
}
let mut cursors = vec![0usize; sub_series.len()];
let mut last_values = vec![0.0f64; sub_series.len()];
all_timestamps
.into_iter()
.map(|ts| {
for (i, points) in sub_series.iter().enumerate() {
while cursors[i] < points.len() && points[cursors[i]].0 <= ts {
last_values[i] = points[cursors[i]].1;
cursors[i] += 1;
}
}
let sum: f64 = last_values.iter().sum();
let value = if is_avg { sum / last_values.len() as f64 } else { sum };
(ts, value)
})
.collect()
}
fn ottyl_reserved_numeric_value_fn(metric_name: &str) -> Option<fn(u64, i64) -> f64> {
match metric_name {
"ottyl_seq" => Some(|sequence, _timestamp_ms| sequence as f64),
"ottyl_ts" => Some(|_sequence, timestamp_ms| timestamp_ms as f64),
_ => None,
}
}
impl EngineSnapshot {
pub(crate) fn evaluate_promql(
&self,
parsed: &PromqlQuery,
start_ms: Option<i64>,
end_ms: Option<i64>,
) -> Result<Vec<PromqlSeriesDto>, String> {
if parsed.metric_name == "ottyl_fmt" {
return Err(
"'ottyl_fmt' is a categorical value (json/logfmt/none), not a number, so it can't be \
used in a dashboard PromQL query; filter on it in the search bar instead, e.g. \
ottyl_fmt==json"
.to_string(),
);
}
if ottyl_reserved_numeric_value_fn(&parsed.metric_name).is_none()
&& !self.index.contains_key(&IndexKey::MetricDeclaration { name: parsed.metric_name.clone() })
&& self.bucket_values(&parsed.metric_name, None).next().is_none()
{
return Err(format!(
"unknown metric \"{}\": no declaration or recorded data for it (was it deleted?)",
parsed.metric_name
));
}
if matches!(parsed.agg, Some(AggOp::Sum) | Some(AggOp::Avg))
&& !parsed.inner_rate
&& self.declared_value_kind(&parsed.metric_name) == Some(MetricPrimitiveKind::String)
{
return Err(format!(
"cannot sum/avg a string-typed metric \"{}\"; use count() or a numeric-typed metric",
parsed.metric_name
));
}
let mut series_points: BTreeMap<BTreeMap<String, String>, Vec<(i64, f64)>> = BTreeMap::new();
if let Some(reserved_value) = ottyl_reserved_numeric_value_fn(&parsed.metric_name) {
let no_labels: BTreeMap<String, MetricPrimitive> = BTreeMap::new();
if matches_label_matchers(&no_labels, &parsed.matchers) {
for (sequence, timestamp, _text) in self.logs_range() {
let timestamp_ms = timestamp.as_millis();
if start_ms.is_none_or(|start| timestamp_ms >= start)
&& end_ms.is_none_or(|end| timestamp_ms <= end)
{
series_points
.entry(BTreeMap::new())
.or_default()
.push((timestamp_ms, reserved_value(sequence, timestamp_ms)));
}
}
}
} else {
for (labels, (_, timestamp, value)) in self.bucket_values(&parsed.metric_name, None)
.filter(|(labels, _)| matches_label_matchers(labels, &parsed.matchers))
.filter(|(_, (_, timestamp, _))| {
start_ms.is_none_or(|start| timestamp.as_millis() >= start)
&& end_ms.is_none_or(|end| timestamp.as_millis() <= end)
})
{
let display_labels: BTreeMap<String, String> =
labels.iter().map(|(name, value)| (name.clone(), value.to_string())).collect();
series_points
.entry(display_labels)
.or_default()
.push((timestamp.as_millis(), coerce_to_f64(value)));
}
}
for points in series_points.values_mut() {
points.sort_by_key(|(timestamp_ms, _)| *timestamp_ms);
}
Ok(match parsed.agg {
None => series_points
.into_iter()
.map(|(labels, points)| PromqlSeriesDto { labels, points })
.collect(),
Some(AggOp::Count) => {
let mut groups: BTreeMap<BTreeMap<String, String>, Vec<Vec<(i64, f64)>>> = BTreeMap::new();
for (labels, points) in series_points {
let count_points: Vec<(i64, f64)> =
points.iter().map(|(timestamp_ms, _)| (*timestamp_ms, 1.0)).collect();
let group_key = project_labels(&labels, &parsed.by_labels);
groups.entry(group_key).or_default().push(count_points);
}
groups
.into_iter()
.map(|(group_labels, sub_series)| {
let points = merge_series_windowed(&sub_series, false);
PromqlSeriesDto { labels: group_labels, points }
})
.collect()
}
Some(AggOp::Sum) | Some(AggOp::Avg) => {
let is_avg = parsed.agg == Some(AggOp::Avg);
let rate_range_ms = parsed
.inner_rate
.then(|| parsed.range.expect("inner rate() always carries a range").as_millis() as i64);
let mut groups: BTreeMap<BTreeMap<String, String>, Vec<Vec<(i64, f64)>>> = BTreeMap::new();
for (labels, points) in series_points {
let points = match rate_range_ms {
Some(range_ms) => windowed_rate_series(&points, range_ms, start_ms, end_ms),
None => points,
};
let group_key = project_labels(&labels, &parsed.by_labels);
groups.entry(group_key).or_default().push(points);
}
groups
.into_iter()
.map(|(group_labels, sub_series)| {
let points = if rate_range_ms.is_some() {
merge_series_locf(&sub_series, is_avg)
} else {
merge_series_windowed(&sub_series, is_avg)
};
PromqlSeriesDto { labels: group_labels, points }
})
.collect()
}
Some(AggOp::Rate) => {
let range_ms = parsed.range.expect("rate() always carries a range").as_millis() as i64;
series_points
.into_iter()
.map(|(labels, points)| PromqlSeriesDto {
labels,
points: windowed_rate_series(&points, range_ms, start_ms, end_ms),
})
.collect()
}
Some(AggOp::Gauge) => {
series_points
.into_iter()
.filter_map(|(labels, points)| {
let last = *points.last()?;
Some(PromqlSeriesDto { labels, points: vec![last] })
})
.collect()
}
Some(AggOp::Histogram) => {
let bucket_width =
parsed.bucket_width.expect("histogram() always carries a bucket width");
let last_ts = series_points
.values()
.flat_map(|points| points.iter().map(|(ts, _)| *ts))
.max()
.unwrap_or(0);
let mut groups: BTreeMap<BTreeMap<String, String>, Vec<&(i64, f64)>> = BTreeMap::new();
for (labels, points) in &series_points {
let group_key = project_labels(labels, &parsed.by_labels);
groups.entry(group_key).or_default().extend(points.iter());
}
groups
.into_iter()
.flat_map(|(group_labels, points)| {
let mut counts: BTreeMap<i64, u64> = BTreeMap::new();
for (_, value) in points {
let bucket_index = (value / bucket_width).floor() as i64;
*counts.entry(bucket_index).or_insert(0) += 1;
}
counts.into_iter().map(move |(bucket_index, count)| {
let upper_bound = (bucket_index + 1) as f64 * bucket_width;
let mut labels = group_labels.clone();
labels.insert("le".to_string(), upper_bound.to_string());
PromqlSeriesDto { labels, points: vec![(last_ts, count as f64)] }
})
})
.collect()
}
})
}
}
src/engine/snapshot/sort.rs
+183
View File
@@ -0,0 +1,183 @@
//! Sort-helper indexes: a per-param `sort_by` lets the logs view return results ordered by a
//! declared param's value instead of ingest order. Building that ordering from scratch is
//! O(n log n) over every log, so it's memoized in `EngineSnapshot::sort_helper_indexes` and
//! kept incrementally up to date on ingest, rather than rebuilt per query.
use crate::consts::SORT_HELPER_INDEX_PURGE_AFTER;
use crate::engine::data_value::DataValue;
use crate::engine::index_key::IndexKey;
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::snapshot::metrics::parse_metric_primitive;
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::timestamp::Timestamp;
use regex::Regex;
use std::collections::{BTreeMap, HashMap};
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(crate) enum SortHelperKey {
WithParam(MetricPrimitive, u64),
WithoutParam(u64),
}
#[derive(Debug, Clone)]
pub(crate) struct SortHelperIndexEntry {
pub(crate) index: BTreeMap<SortHelperKey, Timestamp>,
pub(crate) last_accessed: std::time::Instant,
}
pub(super) type CompiledParamDeclaration = (String, Regex, String, Option<String>, MetricPrimitiveKind, bool);
fn derive_single_param(
text: &str,
declarations: &[CompiledParamDeclaration],
param_name: &str,
) -> Option<MetricPrimitive> {
let (_, regex, capture_group, json_field_path, kind, log_stream_gated) =
declarations.iter().find(|(name, ..)| name == param_name)?;
if *log_stream_gated && !crate::engine::logfmt::is_log_stream_line(text) {
return None;
}
if *kind == MetricPrimitiveKind::Boolean {
return Some(MetricPrimitive::Boolean(regex.is_match(text)));
}
let captures = regex.captures(text)?;
let raw: String = match json_field_path.as_deref().filter(|path| !path.is_empty()) {
Some(field_path) => crate::engine::data_value::extract_json_field(text, field_path)?,
None => captures.name(capture_group)?.as_str().to_string(),
};
parse_metric_primitive(&raw, *kind)
}
impl EngineSnapshot {
pub(super) fn build_sort_helper_index(&self, sort_by: &str) -> BTreeMap<SortHelperKey, Timestamp> {
let compiled_params = self.compiled_param_declarations();
self.logs_range()
.map(|(sequence, timestamp, text)| {
let key = match derive_single_param(text, &compiled_params, sort_by) {
Some(value) => SortHelperKey::WithParam(value, sequence),
None => SortHelperKey::WithoutParam(sequence),
};
(key, timestamp)
})
.collect()
}
pub(crate) fn update_sort_helper_indexes_on_ingest(&mut self, sequence: u64, timestamp: Timestamp, text: &str) {
if self.sort_helper_indexes.is_empty() {
return;
}
let compiled_params = self.compiled_param_declarations();
for (sort_by, entry) in self.sort_helper_indexes.iter_mut() {
let key = match derive_single_param(text, &compiled_params, sort_by) {
Some(value) => SortHelperKey::WithParam(value, sequence),
None => SortHelperKey::WithoutParam(sequence),
};
entry.index.insert(key, timestamp);
}
}
pub(crate) fn purge_stale_sort_helper_indexes(&mut self) {
self.sort_helper_indexes
.retain(|_, entry| entry.last_accessed.elapsed() < SORT_HELPER_INDEX_PURGE_AFTER);
}
pub(crate) fn ensure_sort_helper_index_built(&mut self, sort_by: &str) {
if !self.sort_helper_indexes.contains_key(sort_by) {
let built = self.build_sort_helper_index(sort_by);
self.sort_helper_indexes.insert(
sort_by.to_string(),
SortHelperIndexEntry { index: built, last_accessed: std::time::Instant::now() },
);
}
let entry = self.sort_helper_indexes.get_mut(sort_by).expect("just inserted above if missing");
entry.last_accessed = std::time::Instant::now();
}
pub(crate) fn compiled_param_declarations(&self) -> Vec<CompiledParamDeclaration> {
self.param_declarations_range()
.filter(|(_, _, _, _, _, elapsed_since_last, _, _)| !elapsed_since_last)
.filter_map(|(name, _, capture_group, json_field_path, kind, _, _, log_stream_gated)| {
self.param_regex_cache.get(name).map(|re| {
(
name.to_string(),
re.clone(),
capture_group.to_string(),
json_field_path.map(|path| path.to_string()),
kind,
log_stream_gated,
)
})
})
.collect()
}
}
pub(crate) fn derive_log_params(text: &str, declarations: &[CompiledParamDeclaration]) -> BTreeMap<String, MetricPrimitive> {
let mut params = BTreeMap::new();
for (name, regex, capture_group, json_field_path, kind, log_stream_gated) in declarations {
if *log_stream_gated && !crate::engine::logfmt::is_log_stream_line(text) {
continue;
}
if *kind == MetricPrimitiveKind::Boolean {
params.insert(name.clone(), MetricPrimitive::Boolean(regex.is_match(text)));
continue;
}
let Some(captures) = regex.captures(text) else {
continue;
};
let raw: String = match json_field_path.as_deref().filter(|path| !path.is_empty()) {
Some(field_path) => match crate::engine::data_value::extract_json_field(text, field_path) {
Some(value) => value,
None => continue,
},
None => match captures.name(capture_group) {
Some(m) => m.as_str().to_string(),
None => continue,
},
};
let Some(value) = parse_metric_primitive(&raw, *kind) else {
continue;
};
params.insert(name.clone(), value);
}
params
}
pub(super) fn log_entry_params(text: &str, declarations: &[CompiledParamDeclaration]) -> BTreeMap<String, MetricPrimitive> {
let mut params = derive_log_params(text, declarations);
params.insert("ottyl_fmt".to_string(), MetricPrimitive::String(crate::engine::logfmt::detect_format(text).to_string()));
params
}
pub(super) enum SortHelperAccess<'a> {
#[cfg(test)]
Mutable(&'a mut HashMap<String, SortHelperIndexEntry>),
Immutable(&'a HashMap<String, SortHelperIndexEntry>),
}
impl<'a> SortHelperAccess<'a> {
#[cfg_attr(not(test), allow(unused_variables))]
pub(super) fn resolve(
self,
index: &BTreeMap<IndexKey, DataValue>,
sort_by: &str,
) -> Option<&'a BTreeMap<SortHelperKey, Timestamp>> {
match self {
#[cfg(test)]
SortHelperAccess::Mutable(map) => {
if !map.contains_key(sort_by) {
let built = EngineSnapshot::from_index(index.clone()).build_sort_helper_index(sort_by);
map.insert(
sort_by.to_string(),
SortHelperIndexEntry { index: built, last_accessed: std::time::Instant::now() },
);
}
let entry = map.get_mut(sort_by).expect("just inserted above if missing");
entry.last_accessed = std::time::Instant::now();
map.get(sort_by).map(|entry| &entry.index)
}
SortHelperAccess::Immutable(map) => map.get(sort_by).map(|entry| &entry.index),
}
}
}
src/engine/tests.rs
+2292
View File
File diff suppressed because it is too large Load Diff
src/timestamp.rs → src/engine/timestamp.rs
-1
View File
@@ -1,7 +1,6 @@
use serde::Serialize; use serde::Serialize;
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
/// Wall-clock instant stored as nanoseconds since the Unix epoch (not `std::time::Instant`, which has no epoch and can't be serialized).
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize)] #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub struct Timestamp(i64); pub struct Timestamp(i64);
src/filter_parser.pest
-48
View File
@@ -1,48 +0,0 @@
// Grammar for the FilterBar query language.
//
// Supports:
// - regex("...") raw-text regex search over log lines
// - metric_name[param] <op> value metric comparisons
// - json[field] <op> value JSON field comparisons (line parsed as JSON)
// - sequence <op> value reserved metric name (no [param])
// - timestamp <op> value reserved metric name (no [param])
// - `and` / `or` boolean combinators, with `and` binding tighter than `or`
// - parenthesized sub-expressions
WHITESPACE = _{ " " | "\t" | "\n" | "\r" }
file = { SOI ~ or_expr ~ EOI }
or_expr = { and_expr ~ (or_kw ~ and_expr)* }
and_expr = { term ~ (and_kw ~ term)* }
term = _{ regex_expr | comparison | "(" ~ or_expr ~ ")" }
// These must be atomic (`@`): a non-atomic rule would let pest's implicit
// whitespace insertion slip in between the keyword and its own boundary
// check below, which breaks the lookahead (it would see the first
// non-whitespace character *after* skipping the separating space, instead
// of the character immediately following the keyword).
and_kw = @{ "and" ~ !(ASCII_ALPHANUMERIC | "_") }
or_kw = @{ "or" ~ !(ASCII_ALPHANUMERIC | "_") }
regex_expr = { "regex" ~ "(" ~ string_literal ~ ")" }
string_literal = ${ "\"" ~ inner_str ~ "\"" }
inner_str = @{ (escaped_char | !("\"" | "\\") ~ ANY)* }
escaped_char = @{ "\\" ~ ANY }
comparison = _{ json_comparison | metric_comparison }
// `json[field] <op> value` must be tried before `metric_comparison` since "json" is
// otherwise a syntactically valid (if never declared) metric name.
json_comparison = { "json" ~ "[" ~ param ~ "]" ~ op ~ value }
metric_comparison = { metric ~ ("[" ~ param ~ "]")? ~ op ~ value }
metric = @{ ident }
param = @{ ident }
ident = @{ (ASCII_ALPHANUMERIC | "_")+ }
op = { "<=" | ">=" | "==" | "!=" | "<" | ">" }
value = @{ (!(WHITESPACE | ")") ~ ANY)+ }
src/fixtures/go_log_sample.txt
+3
View File
@@ -0,0 +1,3 @@
{"time":"2026-09-06T10:35:34.499921+01:00","level":"WARN","msg":"eventC","labelC":1}
{"time":"2026-09-06T10:35:35.000837+01:00","level":"ERROR","msg":"eventB","labelC":2,"labelA":3}
{"time":"2026-09-06T10:35:36.504236+01:00","level":"INFO","msg":"eventB","labelB":"three"}
src/fixtures/log_stream_sample.txt
+697
View File
@@ -0,0 +1,697 @@
Timestamp Thread Type Activity PID TTL
2026-09-06 01:48:54.465144+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:54.466387+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.196298]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:54.466396+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.196300]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:54.466397+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.196305]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:54.466397+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.196308]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:54.466386+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.496702+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:54.500366+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) [com.apple.Multitouch:Plugin] [TP] Dispatching digitizer event with 3 children, _eventMask=0x0 _childEventMask=0x2 Cancel=0 Touching=1 inRange=1
2026-09-06 01:48:54.532524+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.554814+0100 0x144b Default 0x0 606 7 WindowServer: (SkyLight) [com.apple.SkyLight:KeyboardEvent] kCGSEventKeyUp was part of a sequence with destination <keyboardFocus; pid: 1343; token: viewbridge-key-window>
2026-09-06 01:48:54.555315+0100 0x2e72 Default 0x0 1343 0 kitty: (TextInputUIMacHelper) [com.apple.TextInputUI:CursorUI] -[TUINSCursorUIController scheduleUpdateCursorLocation]
2026-09-06 01:48:54.562153+0100 0x3d1 Default 0x0 0 0 kernel: (AppleAVD) AppleAVD: INFO: logPerfAndUserClientStats(): core 0, Vmin/nom/max: 99% 0% 0%, clientCount 1 1 1 1 1
2026-09-06 01:48:54.564722+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.598759+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.632768+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.666589+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.671971+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Driver Event: <airport[642]> _bsd_80211_event_callback: APPLE80211_M_WEIGHT_AVG_LQM_UPDATE/145 (en0)
2026-09-06 01:48:54.680948+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:54.700119+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.732678+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.766150+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.801202+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.807884+0100 0x4e1fd2 Activity 0x9cc91e 975 0 CoreServicesUIAgent: Handle XPC Request
2026-09-06 01:48:54.807881+0100 0x4e214c Activity 0x9cc91d 975 0 CoreServicesUIAgent: Handle XPC Request
2026-09-06 01:48:54.807903+0100 0x4e23c3 Activity 0x9cc91f 975 0 CoreServicesUIAgent: Handle XPC Request
2026-09-06 01:48:54.808141+0100 0x4e23c4 Error 0x9cc91e 975 2 CoreServicesUIAgent: [com.apple.launchservices:uiagent] CSUIGetDisplayName, failing with -10810 for path '<private>
2026-09-06 01:48:54.808194+0100 0x4e23c4 Error 0x9cc91f 975 2 CoreServicesUIAgent: [com.apple.launchservices:uiagent] CSUIGetDisplayName, failing with -10810 for path '<private>
2026-09-06 01:48:54.808238+0100 0x4e23c4 Error 0x9cc91d 975 2 CoreServicesUIAgent: [com.apple.launchservices:uiagent] CSUIGetDisplayName, failing with -10810 for path '<private>
2026-09-06 01:48:54.823648+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255552387 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:54.823667+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:54.832643+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.845410+0100 0x4dff68 Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.LE.Scan] Matched UUID 0xFCB2 for device "A1161273-82A9-54D0-9283-720AC7DD464C"
2026-09-06 01:48:54.845529+0100 0x4dff68 Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.LE.Scan] Matched UUID 0xFCB2 for device "A1161273-82A9-54D0-9283-720AC7DD464C"
2026-09-06 01:48:54.845773+0100 0x4df240 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: got avenger advertisement", "subHarvester":"Avenger"}
2026-09-06 01:48:54.845815+0100 0x4defc1 Default 0x0 572 0 locationd: (SPFinder) [com.apple.icloud.SPFinder:advertisementCache] saveBeaconPayload: 1
2026-09-06 01:48:54.845818+0100 0x4df240 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"convertToSPAdvertisement", "address":<private>, "data":<private>, "date":<private>, "rssi":<private>, "status":<private>, "reserved":<private>, "subHarvester":"Avenger"}
2026-09-06 01:48:54.845837+0100 0x4defc1 Activity 0x9d0e2a 572 0 locationd: (SPFinder) SPAdvertisementCache saveBeaconPayloads:completion:
2026-09-06 01:48:54.845847+0100 0x4df240 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: advertisement is near-owner from other people and do not process it", "subHarvester":"Avenger"}
2026-09-06 01:48:54.846284+0100 0x4dfe20 Activity 0x9d0dbf 746 0 searchpartyd: (FindMyBase) SaveObservedAdvertisment
2026-09-06 01:48:54.846660+0100 0x4dfe20 Default 0x9d0dbf 746 0 searchpartyd: (FindMyBase) [com.apple.icloud.searchpartyd:observationStoring] TIME: ObservationStoring.Reconcile: <private>
2026-09-06 01:48:54.865403+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.889179+0100 0x4de289 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:54.899419+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.932587+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.940343+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:54.965716+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:54.969316+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:54.999946+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.002756+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10141] : Active Sockets false ValidSvc 4 NumAirplay 0 AFHandlePending 0 TimeOfLastAFHandleRequest 141710 ms
2026-09-06 01:48:55.002786+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10239]: BonJourTrig 1 ValidSvc 4 RTApp 0 TSReq 0 HasActAirDrop 0 SocketsActive 0
2026-09-06 01:48:55.002799+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setScheduleState[11041]: reason:UserTriggered sc:Infra Priority and force:NO, AWDL-restore:No
2026-09-06 01:48:55.002824+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: AWDLStateDump: AWDL up time: 15732 Secs, 82:45:A3:D5:31:34 peers:0, cached:0(0), rtg(0), <SDB: :(00:00:00:00:00:00) 44, infraRT:Yes, country:GB>
2026-09-06 01:48:55.002862+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: AWDLStateDump: Services[Valid 4 Airplay 0 WiFid 0 AirDrop 0 nonAirplay:4 rdlink:0 clink:0]
2026-09-06 01:48:55.002872+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: AWDLStateDump: AirPlay:0 AirPlayToHT:0 AirPlayOverInfra:0 SideCar:0 SideCarSender:0 SideCarPeer:0x0 TimeSync:0 HighThroughput:0
2026-09-06 01:48:55.002892+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: AWDLStateDump: AWDL enabled with no data connection from 15732 seconds
2026-09-06 01:48:55.002901+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: AWDLStateDump: PSFDwell Support/Active: 1/0 {0ms 0ms}, miconfigured:0 sessin count:7, MIoNDwell:35, dwelEvent:0
2026-09-06 01:48:55.002922+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: AWDL ON:[infra(44) 90%] [sdb:0] (6/44/44) [44 0 44 0 0 0 0 0 6 0 44 0 0 0 0 0] Infra Priority
2026-09-06 01:48:55.002931+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.stats.awdl0: -------------------------------------------------------------------------------------------------------------
2026-09-06 01:48:55.002941+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.stats.awdl0: AFSPerSlot : [0 0 0 0] [0 0 0 0] [0 0 0 0] [0 0 0 0]
2026-09-06 01:48:55.002945+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.stats.awdl0: AFsPerSignal (batchSize -1) : 0 <= 1, 0 <= 10, 0 <= 30, 0 <= 50, 0 <= 100, 0 > 100
2026-09-06 01:48:55.002949+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.stats.awdl0: TimePerSignal : 0 < 250uSec, 0 250-500uSec, 0 500-1000uSec, 0 1000-2000uSec, 0 > 2000uSec
2026-09-06 01:48:55.002951+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.stats.awdl0: TimePerAF : 0 < 100uSec, 0 100-250uSec, 0 250-500uSec, 0 500-1000uSec, 0 > 1000uSec
2026-09-06 01:48:55.002954+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.stats.awdl0: -------------------------------------------------------------------------------------------------------------
2026-09-06 01:48:55.002965+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.wifi.awdl0: LQM-WiFi:AWDL State #16 Infra Priority(29) DutyCycle 10 StateDuration 45lu.791s[255506.775 - 0.000] StateComplete 0
2026-09-06 01:48:55.002969+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.wifi.awdl0: LQM-WiFi:AWDL Active Time 500ms(9%) 5.002s[255547.563s - 255552.566s]
2026-09-06 01:48:55.003060+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) IO80211PeerMonitor::checkForDPS() Enter
2026-09-06 01:48:55.003076+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: MonitorTimer en0/5000: packet counts: I239 Q0 O295
2026-09-06 01:48:55.003089+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: postMessageOnInfraInterface:3463 posting msgId:169 on skywalk infraInterface
2026-09-06 01:48:55.003106+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) LQM-WiFi: BE : Count: 292 avgLatencyMs:3 maxLatencyMs:7
2026-09-06 01:48:55.003109+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) LQM-WiFi: BK : Count: 2 avgLatencyMs:1 maxLatencyMs:1
2026-09-06 01:48:55.003110+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) LQM-WiFi: VI : Count: 0 avgLatencyMs:0 maxLatencyMs:0
2026-09-06 01:48:55.003112+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) LQM-WiFi: VO : Count: 1 avgLatencyMs:1 maxLatencyMs:1
2026-09-06 01:48:55.020289+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:55.032976+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.056189+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:55.066877+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.099602+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.134018+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.165500+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.914235]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.165510+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.914238]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:55.165510+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.914246]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.165500+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.200023+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:55.200400+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.925162]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.200409+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.925165]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:55.200409+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.925172]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.200409+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.934699]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.200410+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.934702]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:55.200410+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.934708]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:55.200410+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.934712]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.200411+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.934718]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:55.200415+0100 0x4a3b07 Default 0x0 0 0 kernel: [745895.934722]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:55.200398+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.232960+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.236607+0100 0x4e2059 Default 0x0 12260 0 firefox: (Network) [com.apple.network:path] nw_path_libinfo_path_check [C2C8616C-6980-4FDC-9314-BD5FEF18542E Hostname#d4e6e062:0 tcp, legacy-socket, attribution: developer]
libinfo check path: <private>
2026-09-06 01:48:55.236975+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820] DNSServiceQueryRecord START -- qname: <mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='>, qtype: A, flags: 0x1D000, interface index: 0, client pid: 12260 (firefox), name hash: 70429ff7
2026-09-06 01:48:55.237323+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q2661] Question assigned DNS service 84
2026-09-06 01:48:55.237423+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114821] DNSServiceQueryRecord START -- qname: <mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='>, qtype: AAAA, flags: 0x1D000, interface index: 0, client pid: 12260 (firefox), name hash: 70429ff7
2026-09-06 01:48:55.237588+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114821->Q58675] Question assigned DNS service 84
2026-09-06 01:48:55.237601+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q58675] DetermineUnicastQuerySuppression: Query suppressed for <mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='> AAAA (AAAA records are unusable)
2026-09-06 01:48:55.237640+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q58675] GenerateNegativeResponse: Generating negative response for question
2026-09-06 01:48:55.237651+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114821->Q58675] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: 70429ff7, type: AAAA, rdata: <none>
2026-09-06 01:48:55.237903+0100 0x4e172c Default 0x0 639 0 mDNSResponder: [com.apple.mdns:resolver] [Q2661] Sent 55-byte query #1 to <IPv4:BBEFvEFo> over UDP via en0/14 -- id: 0x089F (2207), flags: 0x0100 (Q/Query, RD, NoError), counts: 1/0/0/0, BBgyliWZ IN A?
2026-09-06 01:48:55.254247+0100 0x4e172c Default 0x0 639 0 mDNSResponder: [com.apple.mdns:resolver] [Q2661] Received acceptable 155-byte response from <IPv4:BBEFvEFo> over UDP via en0/14 -- id: 0x089F (2207), flags: 0x8180 (R/Query, RD, RA, NoError), counts: 1/5/0/0, BBgyliWZ IN A?, 45 IN CNAME BBRnzZbz, 34 IN A BBXtrByy, 34 IN A BBEFKvtH, 34 IN A BBOpbcRV, 34 IN A BBcTGmrX
2026-09-06 01:48:55.254376+0100 0x4e2058 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q2661] Handling concluded querier: BBgyliWZ A IN
2026-09-06 01:48:55.254443+0100 0x4e2058 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q2661] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: 70429ff7, type: CNAME, rdata: <mask.hash: 'cDKUsCTEZJB+B/gEdA8N4g=='>
2026-09-06 01:48:55.254463+0100 0x4e2058 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q2661] AnswerQuestionByFollowingCNAME: following CNAME referral -- CNAME: <mask.hash: 'hCeMQekHxjAP1gJ3ZlTg5A=='>, referral count: 0
2026-09-06 01:48:55.254711+0100 0x4e2058 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q8111] Question assigned DNS service 84
2026-09-06 01:48:55.254778+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q8111] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: 4d30d13e, type: A, rdata: <mask.hash: '6b9WEaPm7Gqw4Flcl9+ddQ=='>
2026-09-06 01:48:55.254807+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q8111] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: 4d30d13e, type: A, rdata: <mask.hash: 'TfCnnBV+Z8bdyTgzgKVY6A=='>
2026-09-06 01:48:55.254820+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q8111] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: 4d30d13e, type: A, rdata: <mask.hash: 'I/iOn17GO6bh2gByjdYhqw=='>
2026-09-06 01:48:55.254831+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820->Q8111] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: 4d30d13e, type: A, rdata: <mask.hash: 'J24bRFaq4i4juI0qgCq6/g=='>
2026-09-06 01:48:55.254894+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114820] DNSServiceQueryRecord STOP -- name hash: 4d30d13e, duration: 1s
2026-09-06 01:48:55.254919+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114821] DNSServiceQueryRecord STOP -- name hash: 70429ff7, duration: 1s
2026-09-06 01:48:55.255418+0100 0x11696 Default 0x0 0 0 kernel: [745895.993200]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:55.255424+0100 0x11696 Default 0x0 0 0 kernel: [745895.993205]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:55.255425+0100 0x11696 Default 0x0 0 0 kernel: [745895.993207]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:55.255417+0100 0x11696 Default 0x0 0 0 kernel: SK[6]: flow_entry_alloc fe "0 proc kernel_task(0)firefox nx_port 1 flow_uuid 74BDAEDA-44EF-49A6-9EF0-A23219CFCCE0 flags 0x14120<CONNECTED,QOS_MARKING,EXT_PORT,EXT_FLOWID> ipver=4,src=<IPv4-redacted>.53525,dst=<IPv4-redacted>.443,proto=0x06 mask=0x0000003f,hash=0x7baac1cc tp_proto=0x06"
2026-09-06 01:48:55.255454+0100 0x11696 Default 0x0 0 0 kernel: tcp connect outgoing: [<IPv4-redacted>:53525<-><IPv4-redacted>:443] interface: en0 (skipped: 593)
so_gencnt: 3510228 t_state: SYN_SENT process: firefox:12260 SYN in/out: 0/1 bytes in/out: 0/0 pkts in/out: 0/0 rtt: 0.0 ms rttvar: 250.0 ms base_rtt: 5 ms error: 0 so_error: 0 svc/tc: 0 flow: 0x96139917
2026-09-06 01:48:55.261279+0100 0x1c98 Default 0x0 0 0 kernel: tcp connected: [<IPv4-redacted>:53525<-><IPv4-redacted>:443] interface: en0 (skipped: 593)
so_gencnt: 3510228 t_state: ESTABLISHED process: firefox:12260 SYN in/out: 1/1 bytes in/out: 0/0 pkts in/out: 0/0 rtt: 0.0 ms rttvar: 250.0 ms base_rtt: 5 ms error: 0 so_error: 0 svc/tc: 0 flow: 0x96139917
2026-09-06 01:48:55.267182+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.300375+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.043247]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.300382+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.043252]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:55.300383+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.043256]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.300383+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.043260]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:55.300384+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053765]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300385+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053794]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300386+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053827]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300386+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053860]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300388+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053892]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300388+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053923]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300389+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053958]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300395+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.053990]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300396+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.054022]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:55.300365+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.311658+0100 0x4e2059 Default 0x0 12260 0 firefox: (Network) [com.apple.network:path] nw_path_libinfo_path_check [B2491361-91B6-46C7-842E-F521AA802C01 Hostname#12130b20:0 tcp, legacy-socket, attribution: developer]
libinfo check path: <private>
2026-09-06 01:48:55.311755+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114822] DNSServiceQueryRecord START -- qname: <mask.hash: 'jTFU35oMx3mxTHXGo40ISg=='>, qtype: A, flags: 0x1D000, interface index: 0, client pid: 12260 (firefox), name hash: cc4a6c20
2026-09-06 01:48:55.312122+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114822->Q38154] Question assigned DNS service 84
2026-09-06 01:48:55.312209+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114823] DNSServiceQueryRecord START -- qname: <mask.hash: 'jTFU35oMx3mxTHXGo40ISg=='>, qtype: AAAA, flags: 0x1D000, interface index: 0, client pid: 12260 (firefox), name hash: cc4a6c20
2026-09-06 01:48:55.312387+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114823->Q15486] Question assigned DNS service 84
2026-09-06 01:48:55.312402+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q15486] DetermineUnicastQuerySuppression: Query suppressed for <mask.hash: 'jTFU35oMx3mxTHXGo40ISg=='> AAAA (AAAA records are unusable)
2026-09-06 01:48:55.312448+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q15486] GenerateNegativeResponse: Generating negative response for question
2026-09-06 01:48:55.312465+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114823->Q15486] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: cc4a6c20, type: AAAA, rdata: <none>
2026-09-06 01:48:55.312719+0100 0x4e2058 Default 0x0 639 0 mDNSResponder: [com.apple.mdns:resolver] [Q38154] Sent 53-byte query #1 to <IPv4:BBEFvEFo> over UDP via en0/14 -- id: 0xCBF5 (52213), flags: 0x0100 (Q/Query, RD, NoError), counts: 1/0/0/0, BBmCsKYR IN A?
2026-09-06 01:48:55.329218+0100 0x4e2058 Default 0x0 639 0 mDNSResponder: [com.apple.mdns:resolver] [Q38154] Received acceptable 69-byte response from <IPv4:BBEFvEFo> over UDP via en0/14 -- id: 0xCBF5 (52213), flags: 0x8180 (R/Query, RD, RA, NoError), counts: 1/1/0/0, BBmCsKYR IN A?, 2008 IN A BBYEzGsX
2026-09-06 01:48:55.329361+0100 0x4e172c Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [Q38154] Handling concluded querier: BBmCsKYR A IN
2026-09-06 01:48:55.329425+0100 0x4e172c Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114822->Q38154] DNSServiceQueryRecord result -- event: add, expired: no, ifindex: 0, name hash: cc4a6c20, type: A, rdata: <mask.hash: 'JGEmga7PvGVKnDgblBSHtg=='>
2026-09-06 01:48:55.329646+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114822] DNSServiceQueryRecord STOP -- name hash: cc4a6c20, duration: 0s
2026-09-06 01:48:55.329671+0100 0x1639 Default 0x0 639 0 mDNSResponder: [com.apple.mDNSResponder:Default] [R114823] DNSServiceQueryRecord STOP -- name hash: cc4a6c20, duration: 0s
2026-09-06 01:48:55.329920+0100 0x11696 Default 0x0 0 0 kernel: [745896.057369]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.329927+0100 0x11696 Default 0x0 0 0 kernel: [745896.057375]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:55.329927+0100 0x11696 Default 0x0 0 0 kernel: [745896.065099]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.329927+0100 0x11696 Default 0x0 0 0 kernel: [745896.065103]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.329927+0100 0x11696 Default 0x0 0 0 kernel: [745896.065106]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:55.329928+0100 0x11696 Default 0x0 0 0 kernel: [745896.072609]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.329927+0100 0x11696 Default 0x0 0 0 kernel: [745896.072615]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.329920+0100 0x11696 Default 0x0 0 0 kernel: SK[7]: flow_entry_alloc fe "0 proc kernel_task(0)firefox nx_port 1 flow_uuid 543F6B4B-FE19-4D0E-B3DD-2A550A742A3D flags 0x14120<CONNECTED,QOS_MARKING,EXT_PORT,EXT_FLOWID> ipver=4,src=<IPv4-redacted>.53540,dst=<IPv4-redacted>.443,proto=0x06 mask=0x0000003f,hash=0x7c762c2b tp_proto=0x06"
2026-09-06 01:48:55.329956+0100 0x11696 Default 0x0 0 0 kernel: tcp connect outgoing: [<IPv4-redacted>:53540<-><IPv4-redacted>:443] interface: en0 (skipped: 593)
so_gencnt: 3510252 t_state: SYN_SENT process: firefox:12260 SYN in/out: 0/1 bytes in/out: 0/0 pkts in/out: 0/0 rtt: 0.0 ms rttvar: 250.0 ms base_rtt: 0 ms error: 0 so_error: 0 svc/tc: 0 flow: 0x90dee0b7
2026-09-06 01:48:55.333766+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.336428+0100 0x1c98 Default 0x0 0 0 kernel: tcp connected: [<IPv4-redacted>:53540<-><IPv4-redacted>:443] interface: en0 (skipped: 593)
so_gencnt: 3510252 t_state: ESTABLISHED process: firefox:12260 SYN in/out: 1/1 bytes in/out: 0/0 pkts in/out: 0/0 rtt: 0.0 ms rttvar: 250.0 ms base_rtt: 0 ms error: 0 so_error: 0 svc/tc: 0 flow: 0x90dee0b7
2026-09-06 01:48:55.348649+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: isInfraRealtimePacketThresholdAllowed allowed:1 option:32 threshold:50 noRegistrations:1 cachedPeerCount:0 fastDiscoveryInactive:1 fastDiscoveryOnSince:255552912
2026-09-06 01:48:55.348674+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: currentInfraTrafficType:9277 checking if realtime upgrade required with inputPackets:62 outputPackets:53 packetThreshold:50
2026-09-06 01:48:55.348689+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isSteeringRequired 0 Attempt Count 0
2026-09-06 01:48:55.348696+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isRestartSteeringRequired is false
2026-09-06 01:48:55.348709+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255552912 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:55.348719+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:55.365920+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.381034+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.MacCoex] Posting Bluetooth Status Notification <private>
2026-09-06 01:48:55.399752+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.130195]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:55.399761+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.130199]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.399761+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.130202]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:55.399766+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.141340]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:55.399766+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.141344]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.399766+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.141347]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:55.399750+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.407537+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:55.415091+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:55.433664+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.466363+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.190261]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.466373+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.190271]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.466361+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.499477+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.533288+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.544460+0100 0x4de289 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:55.556747+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:55.566993+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.600082+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.632344+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.666250+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.679793+0100 0x1f39 Default 0x0 1018 3 sharingd: (CoreUtils) [com.apple.sharing:SFBLEScanner] [NearbyInfo] Clearing duplicate cache
2026-09-06 01:48:55.679857+0100 0x1f39 Default 0x0 1018 0 sharingd: (WirelessProximity) [com.apple.bluetooth:WirelessProximity] Nearby clear duplicate filter cache type 1
2026-09-06 01:48:55.679911+0100 0x1f39 Default 0x0 1018 3 sharingd: (CoreUtils) [com.apple.sharing:SFBLEScanner] [NearbyAction] Clearing duplicate cache
2026-09-06 01:48:55.679932+0100 0x1f39 Default 0x0 1018 0 sharingd: (WirelessProximity) [com.apple.bluetooth:WirelessProximity] Nearby clear duplicate filter cache type 0
2026-09-06 01:48:55.680056+0100 0x1f39 Default 0x0 1018 3 sharingd: (CoreUtils) [com.apple.sharing:SDNearbyAgentCore] Identification removed stale 27b8d190-4d1a-f393-5364-ee9e347ff0ff
2026-09-06 01:48:55.680234+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: (WPDaemon) [com.apple.bluetooth:WirelessProximity] clear dup filter cache for process sharingd (1018) with scan request of type 16, blob: {length = 0, bytes = 0x}, mask {length = 0, bytes = 0x}, active: 0, duplicates: 0, screen on: No Scan, screen off: No Scan, locked: 1, rssi: -90, peers: (
) nearby scan mode: 10, advbuf: 0, priority critical: 0, range: 0, usecases:
2026-09-06 01:48:55.680281+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: (WPDaemon) [com.apple.bluetooth:WirelessProximity] About to clearDuplicateFilterCache for type: 16
2026-09-06 01:48:55.680354+0100 0x4dbefc Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.XPC] Received XPC message "CBMsgIdClearDuplicateFilterCache" from session "com.apple.bluetoothd-central-589-1"
2026-09-06 01:48:55.680398+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: (WPDaemon) [com.apple.bluetooth:WirelessProximity] clear dup filter cache for process sharingd (1018) with scan request of type 15, blob: {length = 0, bytes = 0x}, mask {length = 0, bytes = 0x}, active: 0, duplicates: 0, screen on: No Scan, screen off: No Scan, locked: 1, rssi: -90, peers: (
) nearby scan mode: 10, advbuf: 0, priority critical: 0, range: 0, usecases:
2026-09-06 01:48:55.680422+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: (WPDaemon) [com.apple.bluetooth:WirelessProximity] About to clearDuplicateFilterCache for type: 15
2026-09-06 01:48:55.684985+0100 0x4dbefc Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.App] clearDuplicateFilterCache for session "com.apple.bluetoothd" type 16
2026-09-06 01:48:55.685072+0100 0x4dbefc Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.XPC] Received XPC message "CBMsgIdClearDuplicateFilterCache" from session "com.apple.bluetoothd-central-589-1"
2026-09-06 01:48:55.688823+0100 0x4dbefc Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.App] clearDuplicateFilterCache for session "com.apple.bluetoothd" type 15
2026-09-06 01:48:55.699243+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.719580+0100 0x4e1c4e Default 0x0 1376 0 WallpaperAerialsExtension: (MediaToolbox) [com.apple.coremedia:] <<<< FigVideoQueue >>>> FigVideoQueueGMStats: (0x86f1eb400) 0 frames enqueued in the last 6 seconds; 0 were suspended. sbuf queue contains 0 frames (0.000 sec), max PTS: nan; LWM: 1.000 HWM: 2.000. Timebase time 238.895, rate 0.00. (pid 1376)
2026-09-06 01:48:55.728451+0100 0x4e1c4e Default 0x0 1376 0 WallpaperAerialsExtension: (MediaToolbox) [com.apple.coremedia:] <<<< FigVideoQueue >>>> FigVideoQueueGMStats: (0x86f1eb800) 0 frames enqueued in the last 6 seconds; 0 were suspended. sbuf queue contains 0 frames (0.000 sec), max PTS: nan; LWM: 1.000 HWM: 2.000. Timebase time 238.895, rate 0.00. (pid 1376)
2026-09-06 01:48:55.728480+0100 0x4e1c50 Default 0x0 1376 0 WallpaperAerialsExtension: (MediaToolbox) [com.apple.coremedia:] <<<< FigVideoQueue >>>> FigVideoQueueGMStats: (0x86f1ea800) 0 frames enqueued in the last 6 seconds; 0 were suspended. sbuf queue contains 0 frames (0.000 sec), max PTS: nan; LWM: 1.000 HWM: 2.000. Timebase time 238.895, rate 0.00. (pid 1376)
2026-09-06 01:48:55.728505+0100 0x4e23d2 Default 0x0 1376 0 WallpaperAerialsExtension: (MediaToolbox) [com.apple.coremedia:] <<<< FigVideoQueue >>>> FigVideoQueueGMStats: (0x86f1ebc00) 0 frames enqueued in the last 6 seconds; 0 were suspended. sbuf queue contains 0 frames (0.000 sec), max PTS: nan; LWM: 1.000 HWM: 2.000. Timebase time 238.895, rate 0.00. (pid 1376)
2026-09-06 01:48:55.731950+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.736436+0100 0x4defc2 Default 0x0 572 0 locationd: (SPFinder) [com.apple.icloud.SPFinder:advertisementCache] saveBeaconPayload: 1
2026-09-06 01:48:55.736480+0100 0x4df240 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: got avenger advertisement", "subHarvester":"Avenger"}
2026-09-06 01:48:55.736490+0100 0x4defc2 Activity 0x9d0e2b 572 0 locationd: (SPFinder) SPAdvertisementCache saveBeaconPayloads:completion:
2026-09-06 01:48:55.736553+0100 0x4df240 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"convertToSPAdvertisement", "address":<private>, "data":<private>, "date":<private>, "rssi":<private>, "status":<private>, "reserved":<private>, "subHarvester":"Avenger"}
2026-09-06 01:48:55.736607+0100 0x4df240 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: advertisement is near-owner from other people and do not process it", "subHarvester":"Avenger"}
2026-09-06 01:48:55.736837+0100 0x4dfe20 Activity 0x9d0ff0 746 0 searchpartyd: (FindMyBase) SaveObservedAdvertisment
2026-09-06 01:48:55.737231+0100 0x4dfe20 Default 0x9d0ff0 746 0 searchpartyd: (FindMyBase) [com.apple.icloud.searchpartyd:observationStoring] TIME: ObservationStoring.Reconcile: <private>
2026-09-06 01:48:55.765790+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.799736+0100 0x4e0051 Activity 0x9d0fd0 553 0 mds: (CoreDuetContext) CoreDuet: ClientContext objectForContextualKeyPath:
2026-09-06 01:48:55.801580+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.824926+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:55.830347+0100 0x4e1ff6 Error 0x0 0 0 kernel: (Sandbox) Sandbox: spotlightknowledged(19183) deny(1) mach-lookup com.apple.diagnosticd
2026-09-06 01:48:55.831847+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.841825+0100 0x4db1a8 Default 0x0 1157 0 mediaanalysisd: Running task MADServiceTextProcessingTask (57542) [QoS: 21]
2026-09-06 01:48:55.841897+0100 0x4db1a8 Default 0x0 1157 0 mediaanalysisd: Watchdog already enabled
2026-09-06 01:48:55.841929+0100 0x4db1a8 Default 0x0 1157 5 mediaanalysisd: [com.apple.mediaanalysis:persistent] [Scheduler] Running <private> (57542) [QoS: 21 Cost: 0.100]; remaining resource: 0.90
2026-09-06 01:48:55.841976+0100 0x4db1a8 Default 0x0 1157 5 mediaanalysisd: [com.apple.mediaanalysis:persistent] [QueuingScheduler] Scheduled foreground task MADServiceTextProcessingTask (MADRequestID 377, VCPMADTaskID 57542) for client <private>
2026-09-06 01:48:55.842274+0100 0x4d2d86 Default 0x0 1157 0 mediaanalysisd: (CoreSceneUnderstanding) Setting new context length 512
2026-09-06 01:48:55.860531+0100 0x4d2d86 Default 0x0 1157 0 mediaanalysisd: (MIL) MMapped <private> to base address 100d80000 with size 2008
2026-09-06 01:48:55.866558+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.872246+0100 0x4de289 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255553435 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:55.872263+0100 0x4de289 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:55.875121+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:55.881387+0100 0x4d2d86 Default 0x0 1157 0 mediaanalysisd: (MIL) MMapped <private> to base address 10c840000 with size 677474
2026-09-06 01:48:55.899719+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.934529+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.673815]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.934541+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.673820]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:55.934542+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.673825]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.934527+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.937693+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:55.966206+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:55.966800+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:55.992284+0100 0x35de04 Default 0x0 0 0 kernel: [745896.723885]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:55.992293+0100 0x35de04 Default 0x0 0 0 kernel: [745896.723889]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.992294+0100 0x35de04 Default 0x0 0 0 kernel: [745896.723892]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:55.992294+0100 0x35de04 Default 0x0 0 0 kernel: [745896.732976]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:55.992294+0100 0x35de04 Default 0x0 0 0 kernel: [745896.732979]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:55.992294+0100 0x35de04 Default 0x0 0 0 kernel: [745896.732987]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:55.992282+0100 0x35de04 Default 0x0 0 0 kernel: process_is_plugin_host: running binary "sh" in keys-off mode due to identity: com.apple.sh
2026-09-06 01:48:55.993293+0100 0x4e23d7 Default 0x0 0 0 kernel: sh[40634] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0xe6c827f1353fb483. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:55.994465+0100 0x4e23d7 Default 0x0 0 0 kernel: process_is_plugin_host: running binary "bash" in keys-off mode due to identity: com.apple.bash
2026-09-06 01:48:55.995238+0100 0x4e23d8 Default 0x0 0 0 kernel: bash[40634] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0x797c4c4fe647b2b9. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:56.000882+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10141] : Active Sockets false ValidSvc 4 NumAirplay 0 AFHandlePending 0 TimeOfLastAFHandleRequest 142708 ms
2026-09-06 01:48:56.000916+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10239]: BonJourTrig 1 ValidSvc 4 RTApp 0 TSReq 0 HasActAirDrop 0 SocketsActive 0
2026-09-06 01:48:56.000928+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setScheduleState[11041]: reason:UserTriggered sc:Infra Priority and force:NO, AWDL-restore:No
2026-09-06 01:48:56.001188+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.001370+0100 0x4e23d9 Default 0x0 0 0 kernel: python3.13[40634] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0x38801cc981e88d6f. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:56.032459+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.772342]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:56.032468+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.772347]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:56.032468+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.772353]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:56.032469+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.772356]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:56.032458+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.049546+0100 0x35de04 Default 0x0 0 0 kernel: process_is_plugin_host: running binary "zsh" in keys-off mode due to identity: com.apple.zsh
2026-09-06 01:48:56.050416+0100 0x4e23da Default 0x0 0 0 kernel: zsh[40635] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0xe6c827f1353fb483. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:56.052423+0100 0x4e23da Activity 0x9d1000 40635 0 zsh: (libsystem_info.dylib) Retrieve User by ID
2026-09-06 01:48:56.056832+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:56.066425+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.806256]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:56.066433+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.806262]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:56.066423+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.068814+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:56.070421+0100 0x4e23de Default 0x0 0 0 kernel: 2.1.261[40637] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0xddbc6b7757294f33. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:56.100262+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.832882]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:56.100272+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.832886]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:56.100273+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.832889]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:56.100273+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.843348]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:56.100274+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.843354]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:56.100261+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.133625+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.166361+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.200737+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.233953+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.961670]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:56.233962+0100 0x4a3b07 Default 0x0 0 0 kernel: [745896.961676]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:56.233952+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.247221+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:56.267171+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.300298+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.047876]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300305+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.047905]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300307+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.047936]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300307+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.047980]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300309+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048016]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300314+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048045]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300315+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048094]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300317+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048123]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300317+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048155]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300318+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048200]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300319+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048233]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300320+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048263]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:56.300320+0100 0x4a3b07 Default 0x0 0 0 kernel: [745897.048311]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/ad
2026-09-06 01:48:56.300287+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.333609+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.366240+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.396619+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: isInfraRealtimePacketThresholdAllowed allowed:1 option:32 threshold:50 noRegistrations:1 cachedPeerCount:0 fastDiscoveryInactive:1 fastDiscoveryOnSince:255553960
2026-09-06 01:48:56.396649+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: currentInfraTrafficType:9277 checking if realtime upgrade required with inputPackets:106 outputPackets:155 packetThreshold:50
2026-09-06 01:48:56.396662+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isSteeringRequired 0 Attempt Count 0
2026-09-06 01:48:56.396671+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isRestartSteeringRequired is false
2026-09-06 01:48:56.396691+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255553960 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:56.396701+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:56.400278+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.408316+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:56.432865+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.462152+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:56.466080+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.499132+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.506657+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:56.533255+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.556058+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:56.565352+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.591174+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) allConsoleUserIDs <private>
2026-09-06 01:48:56.591400+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private> creating an xpc connection {syncXPC: YES}
2026-09-06 01:48:56.591427+0100 0x4e226a Default 0x0 1165 0 parsecd: (libxpc.dylib) [com.apple.xpc:connection] [0xba30a1b80] activating connection: mach=true listener=false peer=false name=com.apple.apsd
2026-09-06 01:48:56.591483+0100 0x4e1fdb Error 0x0 0 0 kernel: (Sandbox) 2 duplicate reports for Sandbox: spotlightknowledged(19183) deny(1) mach-lookup com.apple.diagnosticd
2026-09-06 01:48:56.591494+0100 0x4e1fdb Error 0x0 0 0 kernel: (Sandbox) Sandbox: parsecd(1165) deny(1) mach-lookup com.apple.apsd
2026-09-06 01:48:56.591677+0100 0x4e226a Default 0x0 1165 0 parsecd: (libxpc.dylib) [com.apple.xpc:connection] [0xba30a1b80] failed to do a bootstrap look-up: xpc_error=[159: Unknown error: 159]
2026-09-06 01:48:56.591714+0100 0x4e226a Default 0x0 1165 0 parsecd: (libxpc.dylib) [com.apple.xpc:connection] [0xba30a1b80] invalidated after a failed init
2026-09-06 01:48:56.591861+0100 0x4e1e08 Error 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:xpc] Connection Invalid for service com.apple.apsd
2026-09-06 01:48:56.591878+0100 0x4e1e08 Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] invalidationHandler called for <private>
2026-09-06 01:48:56.591962+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private> Bad response from apsd: <private>
2026-09-06 01:48:56.592001+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] Daemon connection did not succeed. Is apsd running?
2026-09-06 01:48:56.592065+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private>: Will attempt reconnect in 300.000000 seconds
2026-09-06 01:48:56.592102+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private> disconnected from apsd...
2026-09-06 01:48:56.592111+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private> failing 0 pending outgoing messages.
2026-09-06 01:48:56.592118+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private> calling connectIfNecessary
2026-09-06 01:48:56.592205+0100 0x4e226a Default 0x0 1165 0 parsecd: (ApplePushService) [com.apple.apsd:connection] <private>: Reconnect scheduled, skipping immediate connection to com.apple.apsd
2026-09-06 01:48:56.593107+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:56.599921+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.632882+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.656803+0100 0x4d6c29 Default 0x0 643 0 symptomsd: (SymptomEvaluator) [com.apple.symptomsd:bbh] TCP metrics iteration:270 since 30.00 secs: allflows=44/C=0/R=0/W=0/flows=3/unacked=0/rxbytes=14768/txbytes=4622/rxooo=0/rxdup=0/retx=0
2026-09-06 01:48:56.656853+0100 0x4d6c29 Default 0x0 643 0 symptomsd: (SymptomEvaluator) [com.apple.symptomsd:bbh] TCP progress metrics score: 0, problem ratio: 0.00 (baseline: 0.00)
2026-09-06 01:48:56.667344+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.699360+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.732739+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.766028+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.766434+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:56.800349+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.832644+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.866678+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.897602+0100 0x4dfc8c Activity 0x9d0fd1 553 0 mds: (CoreDuetContext) CoreDuet: ClientContext objectForContextualKeyPath:
2026-09-06 01:48:56.897908+0100 0x4dfc8c Activity 0x9d0fd2 553 0 mds: (CoreDuetContext) CoreDuet: ClientContext objectForContextualKeyPath:
2026-09-06 01:48:56.898157+0100 0x4dfc8c Activity 0x9d0fd3 553 0 mds: (CoreDuetContext) CoreDuet: ClientContext objectForContextualKeyPath:
2026-09-06 01:48:56.898614+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.920820+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255554484 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:56.920848+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:56.923048+0100 0x4d2d86 Default 0x0 1157 5 mediaanalysisd: [com.apple.mediaanalysis:persistent] [Scheduler] Completed <private> (57542) [QoS: 21]
2026-09-06 01:48:56.923306+0100 0x4e1ff6 Error 0x0 0 0 kernel: (Sandbox) Sandbox: spotlightknowledged(19183) deny(1) mach-lookup com.apple.diagnosticd
2026-09-06 01:48:56.927493+0100 0x1f39 Default 0x0 1018 3 sharingd: (CoreUtils) [com.apple.sharing:SFBLEScanner] [Prox] Clearing duplicate cache
2026-09-06 01:48:56.927537+0100 0x1f39 Default 0x0 1018 0 sharingd: (WirelessProximity) [com.apple.bluetooth:WirelessProximity] ProximityPairing clear duplicate cache
2026-09-06 01:48:56.927786+0100 0x4dff68 Default 0x0 589 0 bluetoothd: (WPDaemon) [com.apple.bluetooth:WirelessProximity] clear dup filter cache for process sharingd (1018) with scan request of type 7, blob: {length = 0, bytes = 0x}, mask {length = 0, bytes = 0x}, active: 0, duplicates: 0, screen on: No Scan, screen off: No Scan, locked: 1, rssi: -90, peers: (
) nearby scan mode: 10, advbuf: 0, priority critical: 0, range: 0, usecases:
2026-09-06 01:48:56.927825+0100 0x4dff68 Default 0x0 589 0 bluetoothd: (WPDaemon) [com.apple.bluetooth:WirelessProximity] About to clearDuplicateFilterCache for type: 7
2026-09-06 01:48:56.927883+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.XPC] Received XPC message "CBMsgIdClearDuplicateFilterCache" from session "com.apple.bluetoothd-central-589-1"
2026-09-06 01:48:56.930165+0100 0x4de1c1 Default 0x0 589 0 bluetoothd: [com.apple.bluetooth:Server.App] clearDuplicateFilterCache for session "com.apple.bluetoothd" type 7
2026-09-06 01:48:56.933261+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.966964+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:56.986301+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:57.000034+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.001909+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10141] : Active Sockets false ValidSvc 4 NumAirplay 0 AFHandlePending 0 TimeOfLastAFHandleRequest 143709 ms
2026-09-06 01:48:57.001941+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10239]: BonJourTrig 1 ValidSvc 4 RTApp 0 TSReq 0 HasActAirDrop 0 SocketsActive 0
2026-09-06 01:48:57.001953+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setScheduleState[11041]: reason:UserTriggered sc:Infra Priority and force:NO, AWDL-restore:No
2026-09-06 01:48:57.015059+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:57.026106+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:57.033034+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.056352+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:57.066726+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.099720+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.118079+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:57.134134+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.166618+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.196253+0100 0x4df240 Default 0x0 572 0 locationd: (SPFinder) [com.apple.icloud.SPFinder:advertisementCache] saveBeaconPayload: 1
2026-09-06 01:48:57.196249+0100 0x4defc1 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: got avenger advertisement", "subHarvester":"Avenger"}
2026-09-06 01:48:57.196286+0100 0x4df240 Activity 0x9d0e2c 572 0 locationd: (SPFinder) SPAdvertisementCache saveBeaconPayloads:completion:
2026-09-06 01:48:57.196317+0100 0x4defc1 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"convertToSPAdvertisement", "address":<private>, "data":<private>, "date":<private>, "rssi":<private>, "status":<private>, "reserved":<private>, "subHarvester":"Avenger"}
2026-09-06 01:48:57.196359+0100 0x4defc1 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: advertisement is near-owner from other people and do not process it", "subHarvester":"Avenger"}
2026-09-06 01:48:57.196618+0100 0x4dfe20 Activity 0x9d0ff1 746 0 searchpartyd: (FindMyBase) SaveObservedAdvertisment
2026-09-06 01:48:57.197020+0100 0x4e069c Default 0x9d0ff1 746 0 searchpartyd: (FindMyBase) [com.apple.icloud.searchpartyd:observationStoring] TIME: ObservationStoring.Reconcile: <private>
2026-09-06 01:48:57.201156+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.231679+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.266391+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.285883+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:57.300070+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.047797]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300086+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.047821]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300087+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.047850]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300088+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.047904]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300089+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.047934]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300090+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.047960]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300091+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048015]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300091+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048040]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300093+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048071]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300093+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048118]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300095+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048150]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300096+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048184]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:57.300097+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.048237]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/ad
2026-09-06 01:48:57.300059+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.333480+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.366348+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.400255+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.408573+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:57.412060+0100 0x4e2016 Activity 0x9d0fd4 553 0 mds: (CoreDuetContext) CoreDuet: ClientContext objectForContextualKeyPath:
2026-09-06 01:48:57.433241+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.436836+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745898.192389 wlan0.E[819960] [wcl] checkTrafficCounters@2686:realTime before<0> -> after<0>, realTimeNumPackets =<2> inactive before<0> -> after<0>, infraNumPackets =<752> statsIntervalMultiplier=<5>
2026-09-06 01:48:57.436954+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745898.192514 wlan0.E[819961] [wcl] computeAdvisoryScores@666:LQM-WiFi: WiFi Latency: TxLat-hist [3(100 %) / 0(0 %) / 0(0 %) / 0(0 %)] RxLat-hist [0(0 %) / 0(0 %) / 0(0 %) / 0(0 %)]
2026-09-06 01:48:57.437120+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16356
2026-09-06 01:48:57.437176+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16356
2026-09-06 01:48:57.437219+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16360
2026-09-06 01:48:57.437246+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16364
2026-09-06 01:48:57.437279+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16356
2026-09-06 01:48:57.437351+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16360
2026-09-06 01:48:57.437383+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16364
2026-09-06 01:48:57.437270+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16368
2026-09-06 01:48:57.437406+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16368
2026-09-06 01:48:57.437428+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16372
2026-09-06 01:48:57.437450+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16376
2026-09-06 01:48:57.437472+0100 0x4e2150 Default 0x0 929 0 wifip2pd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16380
2026-09-06 01:48:57.438233+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16372
2026-09-06 01:48:57.438397+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16376
2026-09-06 01:48:57.438965+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16380
2026-09-06 01:48:57.439214+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16360
2026-09-06 01:48:57.439350+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16364
2026-09-06 01:48:57.439506+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16368
2026-09-06 01:48:57.439540+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16372
2026-09-06 01:48:57.439564+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16376
2026-09-06 01:48:57.439587+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) Apple80211EventMonitoringHelper:5877 Skipping Padding Read offset is 16380
2026-09-06 01:48:57.439664+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Driver Event: <airport[642]> _bsd_80211_event_callback: APPLE80211_M_REALTIME_SESSION_NOTIFICATION/156 (en0)
2026-09-06 01:48:57.439751+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Info: <airport[642]> 11ax ASR session: new-state = 0x0
2026-09-06 01:48:57.439802+0100 0x4db330 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Info: <airport[642]> airportdProcessRealtimeSessionNotificationEvent channel score: chq = 3, tx-lat = 5, rx-lat = 3, tx-loss = 5, rx-loss = 5, p95-lat = 3, RT = 0x0, link-reco = 0x0
2026-09-06 01:48:57.442603+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745898.198158 wlan0.A[819962] [dk] updateLinkQualityMetrics@750:For current session updated fRxDurMulticast 5892240, fRxDurBroadcast 1909060
2026-09-06 01:48:57.445101+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: isInfraRealtimePacketThresholdAllowed allowed:1 option:32 threshold:50 noRegistrations:1 cachedPeerCount:0 fastDiscoveryInactive:1 fastDiscoveryOnSince:255555008
2026-09-06 01:48:57.445119+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: currentInfraTrafficType:9277 checking if realtime upgrade required with inputPackets:34 outputPackets:35 packetThreshold:50
2026-09-06 01:48:57.445143+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isSteeringRequired 0 Attempt Count 0
2026-09-06 01:48:57.445152+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isRestartSteeringRequired is false
2026-09-06 01:48:57.445166+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255555008 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:57.445176+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:57.449237+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: IO80211PeerManager::reportDataPathEvents APPLE80211_M_RSSI_CHANGED RSSI -68 noise -94 snr 27. Core0Rssi:-66 Core1Rssi:-71
2026-09-06 01:48:57.449468+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Driver Event: <airport[642]> _bsd_80211_event_callback: APPLE80211_M_RSSI_CHANGED/39 (en0)
2026-09-06 01:48:57.449638+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Info: <airport[642]> Per-MLO link stats
2026-09-06 01:48:57.449750+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) airportdProcessTrafficEngineeringEvents: <en0> eventType[39] @[745898.205068], took 0.000171
2026-09-06 01:48:57.449808+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] <airport[642]> airportdProcessTrafficEngineeringEvents: <en0> eventType[39] @[745898.205068], took 0.000171
2026-09-06 01:48:57.450027+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM: rssi=-68dBm per_ant_rssi=(-66dBm, -71dBm) noise=-94dBm snr=27 cca=6.0% ccaSelfTotal=1 ccaOtherTotal=1 interferenceTotal=4 txRate=324.3Mbps txFrames=293 txFail=0 txRetrans=48 txFallbackRate=108.1Mbps rxRate=648.5Mbps rxFrames=322 rxRetryFrames=3 rxToss=0 beaconRecv=49 beaconSched=49 txFwFrames=0 txFwFail=0 txFwRetrans=0 network = <redacted> bssid = <redacted> channel = 44 BW = 80 band = 2
2026-09-06 01:48:57.450274+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: WeightAvgLQM rssi=-69 snr=27 txRate=301662 rxRate=576470
2026-09-06 01:48:57.450293+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: (HighBand) rxCrsGlitch=179 rxBphyCrsGlitch=0 rxStart=2993 rxBadPLCP=18 rxBphyBadPLCP=0 rxBadFCS=6 rxFifo0Ovfl=0 rxFifo1Ovfl=0 rx_nobuf=0 rxAnyErr=1643 rxResponseTimeout=16 rxNoDelim=91 rxFrmTooLong=0 rxFrmTooShort=0
2026-09-06 01:48:57.450303+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: (HighBand) txRTSFrm=169 txRTSFail=0 rxCTSUcast=168 rxRTSUcast=201 txCTSFrm=201 txAMPDU=170 rxBACK=155 txPhyError=0 txAllFrm=763 txMPDU=0 txUcast=170 rxACKUcast=0 OfdmDesense=0 dB
2026-09-06 01:48:57.450311+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: (HighBand) rxBeaconMbss=49 rxBeaconObss=0 rxDataUcastMbss=189 rxMgmtUcastMbss=0 rxCNTRLUcast=549 txACKFrm=0 txBACK=184 ctxFifoFull=0 ctxFifo2Full=0 rxDataMcast=4 rxMgmtMcast=49 rxDrop20s=210
2026-09-06 01:48:57.450316+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: RX AMPDU (HighBand) rxAmpdu=205 txBACK(Ucode)=184 rxMpduInAmpdu=214 rxholes=0 rxdup=0 rxstuck=0 rxoow=0 rxoos=1 rxaddbareq=0 txaddbaresp=0 rxbar=0 txdelba=0 rxdelba=0 rxQueued=1 rxRetryNoBA=0
2026-09-06 01:48:57.450322+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: WME RX MPDUs (rxPER 0 %) in tids 0:292, 1:0, 2:0, 3:0, 4:0, 5:0, 6:0, 7:0
2026-09-06 01:48:57.450328+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: Stats12(5G): bt5g_status=0x3 bt5g_defer_cnt=0 bt5g_no_defer_cnt=0 bt5g_defer_max_switch_dur=0 bt5g_no_defer_max_switch_dur=0 bt5g_switch_succ_cnt=0 bt5g_switch_fail_cnt=0 bt5g_switch_reason_bm=0
2026-09-06 01:48:57.450335+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: Stats12(2G): btDur=10ms(0%) status=0x10d reqTypeMap=0x500000 req=23 gnt=14 abort=2 rx1ovfl=0 latency=0 pmAttempt=0 sucPM=0 sucCTS2S=0 wlanTXPreempt=19 wlanRXPreempt=0 apTXafterPM=0 gntAudio=0 denyAudio=0 gntA2DP=0 denyA2DP=0
2026-09-06 01:48:57.450342+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: (HighBand) rxToss=0 rxLastTossRsn=39 rxNoFrag=0 rxNoCmplId=0 rxNoHaddr=0 rxMulti=2 rxUndec=0
2026-09-06 01:48:57.450354+0100 0x4db330 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] BEGIN REQ [GET INTF NAME] (pid=643 proc=symptomsd bundleID=com.apple.symptomsd codesignID=com.apple.symptomsd service=com.apple.private.corewifi-xpc qos=17 intf=(null) uuid=62638 info=(null))
2026-09-06 01:48:57.450365+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: Stats12(2G): CrtPriCnt=69 PriCnt=88 gntSniff=0 denySniff=0 antGrant0~10ms=157 antGrant10~30ms=0 antGrant30~60ms=0 antGrant>60ms=0 APLeakiness=0 rr_cnt=0 rr_succ_cnt=0
2026-09-06 01:48:57.450398+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: Stats12(2G): btDurAir=97ms(1%) desenseMode=0 wlrssi=0 btrssi=0 profile2gActive=0 profile5gActive=6 inactiveDur=4485 btPmAttemptNoackCnt=0 status2=0x0 [5000ms]
2026-09-06 01:48:57.450406+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: 15.4 Coex Stats V3: Mode=0 Request=0 Grant=0 Duration=0us AckedPM=0 TXedCTS2A=0 GrantDelay=0 CritPhyCal=0 CritRateRecovery=0 CritBcnLoss=0 CritHpp=0 CritBT=0 CritAwdlNan=0 CritMaxDur=1050 CirtAggregate=0 CritBitMapMax=2
2026-09-06 01:48:57.450429+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: 15.4 Coex Stats V3: 154_tx_req_cnt=0 154_rx_req_cnt=0 154_tx_deny_cnt=0 154_rx_deny_cnt=0 [5000ms]
2026-09-06 01:48:57.450417+0100 0x4db330 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] [62638] Incoming QoS is less than 'default', promoting to 'default'
2026-09-06 01:48:57.450437+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: TX(00:00:00:00:00:00) AC<SU MS NB NRS NA CM EX TF FFP MRET FLE> BE<292 0 0 0 0 0 0 0 0 0 0> (5002ms)
2026-09-06 01:48:57.450474+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: TX(00:00:00:00:00:00) AC<SU MS NB NRS NA CM EX TF FFP MRET FLE> BK<2 0 0 0 0 0 0 0 0 0 0> (5002ms)
2026-09-06 01:48:57.450483+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: TX(00:00:00:00:00:00) AC<SU MS NB NRS NA CM EX TF FFP MRET FLE> VI<0 0 0 0 0 0 0 0 0 0 0> (5002ms)
2026-09-06 01:48:57.450504+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: TX(00:00:00:00:00:00) AC<SU MS NB NRS NA CM EX TF FFP MRET FLE> VO<1 0 0 0 0 0 0 0 0 0 0> (5002ms)
2026-09-06 01:48:57.450596+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] LQM-WiFi: L3 Control VO TX(00:00:00:00:00:00) Success=1 NoACK=0 Expired=0 OtherErr=0
2026-09-06 01:48:57.450869+0100 0x4db330 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] END REQ [GET INTF NAME] took 0.000471584s (pid=643 proc=symptomsd bundleID=com.apple.symptomsd codesignID=com.apple.symptomsd service=com.apple.private.corewifi-xpc qos=21 intf=(null) uuid=62638 err=0 reply=<redacted>
2026-09-06 01:48:57.450959+0100 0x4df3b0 Default 0x0 643 0 symptomsd: (SymptomEvaluator) [com.apple.symptomsd:rnf] L2 Metrics on en0: rssi: -68 [-1,-1] -> -68, snr: 27 (cca [wake/total] self/other/intf): [1,1]/[0,1]/[4,4]/6 (txFrames/txReTx/txFail): 293/48/0 -> (was/is) 0/0, txRate: 324.0, rxRate: 648.0, rssiEMA: -67.6, snrEMA: 27.0, txRateEMA: 312.7, rxRateEMA: 534.6, isPrimary: 1, radio: 802.11ax
2026-09-06 01:48:57.451145+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] BEGIN REQ [GET BSSID] (pid=643 proc=symptomsd bundleID=com.apple.symptomsd codesignID=com.apple.symptomsd service=com.apple.private.corewifi-xpc qos=17 intf=(null) uuid=67C47 info=(null))
2026-09-06 01:48:57.451176+0100 0x4dfc53 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] [67C47] Incoming QoS is less than 'default', promoting to 'default'
2026-09-06 01:48:57.451603+0100 0x4dc356 Default 0x0 642 0 airportd: (CoreWiFi) [com.apple.WiFiManager:] [corewifi] END REQ [GET BSSID] took 0.000425417s (pid=643 proc=symptomsd bundleID=com.apple.symptomsd codesignID=com.apple.symptomsd service=com.apple.private.corewifi-xpc qos=21 intf=(null) uuid=67C47 err=0 reply=<redacted>
2026-09-06 01:48:57.467726+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.499311+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.510770+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setSyncFramePeriod:25958 +++ Configuring the sync params(period, length) {16 16}, extlen = 16
2026-09-06 01:48:57.510803+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setSyncFramePeriod::Setting PSF period to 110
2026-09-06 01:48:57.510824+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: programChipWithNewMaster: Programming new Mst: 82:45:A3:D5:31:34 at 255555.074ms HC 0 met 510 my met 510
2026-09-06 01:48:57.510847+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setSyncFramePeriod:25958 +++ Configuring the sync params(period, length) {16 16}, extlen = 16
2026-09-06 01:48:57.510863+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setSyncFramePeriod::Setting PSF period to 110
2026-09-06 01:48:57.510873+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: runElection: ElectionParamsChanged - Prog (F:0, 0) New P:82:45:A3:D5:31:34 R:82:45:A3:D5:31:34 [RLFC:49492 HC:0 RMet:510 SLFC:49492 SMet:510]
2026-09-06 01:48:57.510891+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:57.533659+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.558289+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:57.565927+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.586473+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:57.599662+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.633376+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.641692+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:57.665974+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.699771+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.733070+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.766563+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.800210+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.832349+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.561636]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:57.832356+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.561642]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:57.832347+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.854407+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:57.866599+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.899028+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.932110+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.966597+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:57.969365+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255555533 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:57.969385+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:58.000845+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10141] : Active Sockets false ValidSvc 4 NumAirplay 0 AFHandlePending 0 TimeOfLastAFHandleRequest 144708 ms
2026-09-06 01:48:58.000878+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10239]: BonJourTrig 1 ValidSvc 4 RTApp 0 TSReq 0 HasActAirDrop 0 SocketsActive 0
2026-09-06 01:48:58.000891+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setScheduleState[11041]: reason:UserTriggered sc:Infra Priority and force:NO, AWDL-restore:No
2026-09-06 01:48:58.001361+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.033116+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.034822+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:58.056677+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:58.066963+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.099429+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.132720+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.133661+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:58.139542+0100 0x4e2016 Activity 0x9d0fd5 553 0 mds: (CoreDuetContext) CoreDuet: ClientContext objectForContextualKeyPath:
2026-09-06 01:48:58.166016+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:58.166457+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.200626+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.233134+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.970309]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:58.233143+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.970312]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:58.233143+0100 0x4a3b07 Default 0x0 0 0 kernel: [745898.970319]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:58.233133+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.266457+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.299153+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.047915]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299159+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.047958]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299161+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.047998]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299168+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048043]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299168+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048079]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299169+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048108]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299170+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048147]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299171+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048187]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299171+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048211]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299173+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048265]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299173+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048290]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299174+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048321]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:58.299175+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.048363]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/ad
2026-09-06 01:48:58.299142+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.333347+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.366017+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.400317+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.409677+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:58.432904+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.466670+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.494467+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: isInfraRealtimePacketThresholdAllowed allowed:1 option:32 threshold:50 noRegistrations:1 cachedPeerCount:0 fastDiscoveryInactive:1 fastDiscoveryOnSince:255556058
2026-09-06 01:48:58.494496+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: currentInfraTrafficType:9277 checking if realtime upgrade required with inputPackets:33 outputPackets:32 packetThreshold:50
2026-09-06 01:48:58.494510+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isSteeringRequired 0 Attempt Count 0
2026-09-06 01:48:58.494517+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isRestartSteeringRequired is false
2026-09-06 01:48:58.494532+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255556058 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:58.494541+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:58.498467+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.533601+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.558028+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:58.561375+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:58.566902+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.599663+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.633427+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.665831+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.683749+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:58.690410+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:58.699997+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.732558+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.765115+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.799353+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.832325+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.865970+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.898628+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.933533+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.966260+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:58.989972+0100 0x4defc1 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: got avenger advertisement", "subHarvester":"Avenger"}
2026-09-06 01:48:58.989993+0100 0x4defc2 Default 0x0 572 0 locationd: (SPFinder) [com.apple.icloud.SPFinder:advertisementCache] saveBeaconPayload: 1
2026-09-06 01:48:58.990030+0100 0x4defc2 Activity 0x9d0e2d 572 0 locationd: (SPFinder) SPAdvertisementCache saveBeaconPayloads:completion:
2026-09-06 01:48:58.990047+0100 0x4defc1 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"convertToSPAdvertisement", "address":<private>, "data":<private>, "date":<private>, "rssi":<private>, "status":<private>, "reserved":<private>, "subHarvester":"Avenger"}
2026-09-06 01:48:58.990092+0100 0x4defc1 Default 0x0 572 0 locationd: [com.apple.locationd.Core:Harvester] {"msg":"onAvengerAdvertisementDetected: advertisement is near-owner from other people and do not process it", "subHarvester":"Avenger"}
2026-09-06 01:48:58.990439+0100 0x4e069c Activity 0x9d0ff2 746 0 searchpartyd: (FindMyBase) SaveObservedAdvertisment
2026-09-06 01:48:58.990963+0100 0x4e069c Default 0x9d0ff2 746 0 searchpartyd: (FindMyBase) [com.apple.icloud.searchpartyd:observationStoring] TIME: ObservationStoring.Reconcile: <private>
2026-09-06 01:48:59.000898+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.002606+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10141] : Active Sockets false ValidSvc 4 NumAirplay 0 AFHandlePending 0 TimeOfLastAFHandleRequest 145710 ms
2026-09-06 01:48:59.002639+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLState[10239]: BonJourTrig 1 ValidSvc 4 RTApp 0 TSReq 0 HasActAirDrop 0 SocketsActive 0
2026-09-06 01:48:59.002652+0100 0x4df69c Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: setScheduleState[11041]: reason:UserTriggered sc:Infra Priority and force:NO, AWDL-restore:No
2026-09-06 01:48:59.017950+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255556581 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:59.017974+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:59.031972+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.056221+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:59.066719+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.081521+0100 0x4e20a1 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:59.081758+0100 0x4dfe54 Default 0x0 538 8 powerd: [com.apple.powerd:BDC] Battery percentage last 80 now 79
2026-09-06 01:48:59.082100+0100 0x144b Default 0x0 606 0 WindowServer: (BackBoardHIDEventFoundation) [com.apple.BackBoard:EventDelivery] destinations for Keyboard event: (<keyboardFocus; kitty.1343; token: viewbridge-key-window>)
2026-09-06 01:48:59.082142+0100 0x144b Default 0x0 606 7 WindowServer: (SkyLight) [com.apple.SkyLight:KeyboardEvent] delivery manager destinations for kCGSEventFlagsChanged event: <keyboardFocus; pid: 1343; token: viewbridge-key-window>
2026-09-06 01:48:59.082554+0100 0x2e72 Default 0x0 1343 0 kitty: (TextInputUIMacHelper) [com.apple.TextInputUI:CursorUI] -[TUINSCursorUIController scheduleUpdateCursorLocation]
2026-09-06 01:48:59.083180+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:59.083489+0100 0x4e002b Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:59.087177+0100 0x4dfe54 Default 0x0 538 8 powerd: [com.apple.powerd:BDC] Sending data to powerlog for group BDC_SBC
2026-09-06 01:48:59.099446+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.110515+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) IO80211ControllerMonitor::setAMPDUstat unhandled kAMPDUStat_ type 14
2026-09-06 01:48:59.110533+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) IO80211ControllerMonitor::setAMPDUstat unhandled kAMPDUStat_ type 13
2026-09-06 01:48:59.110544+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) IO80211ControllerMonitor::setAMPDUstat unhandled kAMPDUStat_ type 18
2026-09-06 01:48:59.115191+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.870747 wlan0.A[819963] [dk] parsePHYEcounterV5@31099: Version = 5, ChanSpec=0xe22a, noise_req=13095, noise_crsbit=8021, cal_counter=4181, rxchain=15, chiptemp=37, femtemp=-50, btcx_mode=0, ltecx_mode=0, weakest_rssi=-74, chan_switc
2026-09-06 01:48:59.115222+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.870782 wlan0.A[819964] [dk] parsePHYEcounterV5@31104: gbd_ofdm_desense=0, gbd_bphy_desense=0, featureflag=96, crsmin_pwr_apply_cnt=33882, preempt_status2=64278, ed_crs_status=0, measurehold=0, ocl_disable_reqs=0, gci_rst_tx= 15463,
2026-09-06 01:48:59.115234+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.870797 wlan0.A[819965] [dk] parsePHYEcounterV5@31109: asym_intf_avg_noise[0/1]= [-98, -98], asym_intf_latest_noise[0/1]=[-98, -98], pwridx_init[0/1]=[35, 37], baseindxval[0/1]=[36, 37]
2026-09-06 01:48:59.115247+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.870809 wlan0.A[819966] [dk] parsePHYEcounterV5@31115: estpwradj[0/1]=[60, 60], ed_threshold[0/1]=[-74, -75], crsmin_pwr_idx[0/1]=[7, 7], crsmin_pwr[0/1]=[-95, -94], noise_level_inst[0/1]=[-96, -95]
2026-09-06 01:48:59.120723+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.876283 wlan0.A[819967] [dk] updateChanSeqCommonStats@3612:P2P chan 44: dur 744287836, entry 18673, crsglitch 9822232, start 132520737, badplcp 763283, badfcs 3170566, f0ov 420, f1ov 577, anyerr 30581679, resptout 1168070, frmtooln 0
2026-09-06 01:48:59.120738+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.876300 wlan0.A[819968] [dk] updateChanSeqCommonStats@3612:P2P chan 6: dur 15735521, entry 1, crsglitch 696648, start 188156, badplcp 6484, badfcs 33786, f0ov 0, f1ov 0, anyerr 23863, resptout 0, frmtooln 0, frmtoosh 7630
2026-09-06 01:48:59.120744+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.876308 wlan0.A[819969] NAV stats: ch 44 (135039794 689529 490534 167738 748 86) ch 6 (132084 1835 1281 106 1750 5)
2026-09-06 01:48:59.120783+0100 0x1bf1 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) 745899.876346 wlan0.A[819970] [dk] printDeltaSliceUcodeRxErrs@1925:(HighBand) rxNotPrimary20s=1245 ucodeLateDrop=0 rxInvalidUcodeLateWrongMACPHY=0 rxLate30us=0 rxLegacyFrmInvalid=0 txSifsErr=0 ooseqMacSusp=0
2026-09-06 01:48:59.132741+0100 0x1551 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Driver Event: <airport[642]> _bsd_80211_event_callback: APPLE80211_M_BT_COEX_MODE_STATS_EVENT/190 (en0)
2026-09-06 01:48:59.133026+0100 0x4dc356 Default 0x0 642 0 airportd: (IO80211) [com.apple.WiFiManager:] Info: <airport[642]> handle airportdProcessBtCoexStatsEvent
2026-09-06 01:48:59.133418+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.165682+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.170964+0100 0x35de04 Default 0x0 0 0 kernel: process_is_plugin_host: running binary "sh" in keys-off mode due to identity: com.apple.sh
2026-09-06 01:48:59.172051+0100 0x4e2411 Default 0x0 0 0 kernel: sh[40639] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0xe6c827f1353fb483. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:59.173175+0100 0x4e2411 Default 0x0 0 0 kernel: [745899.928319]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.173180+0100 0x4e2411 Default 0x0 0 0 kernel: [745899.928329]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:59.173174+0100 0x4e2411 Default 0x0 0 0 kernel: process_is_plugin_host: running binary "bash" in keys-off mode due to identity: com.apple.bash
2026-09-06 01:48:59.174017+0100 0x4e2412 Default 0x0 0 0 kernel: bash[40639] triggered unnest of range 0x1ee000000->0x1f2000000 of DYLD shared region in VM map 0xb2a61f073e8c5e91. While not abnormal for debuggers, this increases system memory footprint almost permanently (until the shared region is re-slid).
2026-09-06 01:48:59.200132+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.949345]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.200143+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.949348]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:59.200144+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.949354]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:59.200145+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.949357]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:59.200145+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.949360]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:59.200126+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.214584+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.en0: set Datapath Open needsTrigger 0
2026-09-06 01:48:59.235340+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.984752]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.235348+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.984754]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:59.235349+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.984760]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:59.235350+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.984763]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:59.235339+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.267868+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.997722]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:59.267880+0100 0x4a3b07 Default 0x0 0 0 kernel: [745899.997727]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:59.267881+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.010921]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.267882+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.010924]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:59.267883+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.010929]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:59.267892+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.020472]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.267892+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.020475]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:59.267892+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.020480]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:59.267894+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.020483]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:59.267894+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.020486]: arm_cpu_init(): cpu 15 online
2026-09-06 01:48:59.267864+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.300117+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.030709]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.300125+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.030712]: arm_cpu_init(): cpu 11 online
2026-09-06 01:48:59.300125+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.030718]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:59.300125+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.030721]: arm_cpu_init(): cpu 14 online
2026-09-06 01:48:59.300126+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046426]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300128+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046472]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300129+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046504]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300129+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046540]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300130+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046572]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300131+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046618]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300133+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046638]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300134+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046668]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300135+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046700]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300141+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046749]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300142+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046778]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300143+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.046809]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrdincr)/0x0 vc/sc/tz/crh/tid:0x2/0x1/0x0/0x0/0x2f size:0x3 piopkt:0x0 user:0x4 userext:0x0 hasdata/has2data:0x0/0x0 dstchildid/dstparentid:0x3f/0x2 srcchildid/srcparentid:0x8ea/0x1d addrl/addrh:0x0/0x0 timestamp:0x0 errorvec:0x10
2026-09-06 01:48:59.300143+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.047460]: Ignored NI0 NI0 niGeneral 2 NO_ACCESS error (0x00000010) cmd/st:0x14(ncrd
2026-09-06 01:48:59.300105+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.333193+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.366943+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.098299]: arm_cpu_init(): cpu 10 online
2026-09-06 01:48:59.366951+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.098302]: arm_cpu_init(): cpu 12 online
2026-09-06 01:48:59.366952+0100 0x4a3b07 Default 0x0 0 0 kernel: [745900.098309]: arm_cpu_init(): cpu 13 online
2026-09-06 01:48:59.366941+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.400043+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.409533+0100 0x2e72 Default 0x0 1343 7 kitty: (SkyLight) [com.apple.SkyLight:default] SLSGetDisplaysWithOpenGLDisplayMask, 3614: Display 0x00000001: Mode 54, (0 x 0) [1728 x 1117], Depth 32 BPP, active
2026-09-06 01:48:59.432119+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.467080+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.499502+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.533324+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.543029+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: isInfraRealtimePacketThresholdAllowed allowed:1 option:32 threshold:50 noRegistrations:1 cachedPeerCount:0 fastDiscoveryInactive:1 fastDiscoveryOnSince:255557106
2026-09-06 01:48:59.543059+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: currentInfraTrafficType:9277 checking if realtime upgrade required with inputPackets:99 outputPackets:144 packetThreshold:50
2026-09-06 01:48:59.543072+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isSteeringRequired 0 Attempt Count 0
2026-09-06 01:48:59.543090+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: peerChannelSteer: PEER_CHANNEL_STEER isRestartSteeringRequired is false
2026-09-06 01:48:59.543104+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9668] : In AW: Tx:0 Rx:0 Prev Tx:0 Rx:0 time since 255557106 indatamode 1 infraPriority 1 cpc 0
2026-09-06 01:48:59.543150+0100 0x4de131 Default 0x0 0 0 kernel: (com.apple.DriverKit-AppleBCMWLAN.dext) wlan0:com.apple.p2p.awdl0: monitorAWDLDataMode[9673] : Ignore changing the state as infraPriority 1 P2P-CPC 0
2026-09-06 01:48:59.565970+0100 0x4a3b07 Error 0x0 0 0 kernel: (IOSurface) SID: 0x0 task: <decode: mismatch for [%p] got [SCALAR private sz:0]> buffer: 0x0 buffer->fClientTask = 0x0 not found
2026-09-06 01:48:59.577680+0100 0x4dfe54 Default 0x0 538 8 powerd: [com.apple.powerd:displayState] DesktopMode check on Battery 0
2026-09-06 01:48:59.578339+0100 0x144b Default 0x0 606 0 WindowServer: (BackBoardHIDEventFoundation) [com.apple.BackBoard:EventDelivery] destinations for Keyboard event: (<keyboardFocus; kitty.1343; token: viewbridge-key-window>)
2026-09-06 01:48:59.578385+0100 0x144b Default 0x0 606 7 WindowServer: (SkyLight) [com.apple.SkyLight:KeyboardEvent] delivery manager destinations for kCGSEventKeyDown event: <keyboardFocus; pid: 1343; token: viewbridge-key-window>
2026-09-06 01:48:59.579704+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices: deviceGeneration=0 (prev=0) displayGeneration=0 (prev=0)
2026-09-06 01:48:59.579746+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices(connectedDevices): returning 0 device(s) to caller: (
)
2026-09-06 01:48:59.579759+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices: deviceGeneration=0 (prev=0) displayGeneration=0 (prev=0)
2026-09-06 01:48:59.579773+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices(devices): returning 0 device(s) to caller: (
)
2026-09-06 01:48:59.580218+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices: deviceGeneration=0 (prev=0) displayGeneration=0 (prev=0)
2026-09-06 01:48:59.580238+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices(connectedDevices): returning 0 device(s) to caller: (
)
2026-09-06 01:48:59.580248+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices: deviceGeneration=0 (prev=0) displayGeneration=0 (prev=0)
2026-09-06 01:48:59.580273+0100 0x2e72 Default 0x0 1343 0 kitty: (SidecarCore) [com.apple.sidecar:core] updateDevices(devices): returning 0 device(s) to caller: (
)
2026-09-06 01:48:59.580663+0100 0x2e72 Default 0x0 1343 0 kitty: (TextInputUIMacHelper) [com.apple.TextInputUI:CursorUI] -[TUINSCursorUIController scheduleUpdateCursorLocation]
src/fixtures/py_log_sample.txt
+3
View File
@@ -0,0 +1,3 @@
2026-09-06 10:35:55,217 INFO delete-py-log event=eventC labelC=1
2026-09-06 10:35:55,627 INFO delete-py-log event=eventB labelB=one
2026-09-06 10:35:56,442 INFO delete-py-log event=eventB labelA=1
src/index_key.rs
-39
View File
@@ -1,39 +0,0 @@
use crate::metric_primitive::MetricPrimitive;
use crate::timestamp::Timestamp;
use serde::Serialize;
use std::collections::BTreeMap;
/// Key type for the engine's primary index. Variant order matters: derived `Ord` compares the discriminant first, carving the map into contiguous ranges in declaration order (logs, params, metric declarations, buckets).
#[allow(dead_code)]
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub enum IndexKey {
/// Marker for the start of the whole range. Never inserted, only navigated to.
Start,
IngestedLog {
sequence: u64,
timestamp: Timestamp,
},
/// Marker between the log range and the param range.
LogParamSplit,
ParamDeclaration {
name: String,
},
/// Marker between the param range and the metric declaration range.
ParamMetricSplit,
MetricDeclaration {
name: String,
},
/// Marker between metric declarations and their buckets.
MetricBucketSplit,
Bucket {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
/// Internal bookkeeping tracking the last event timestamp per metric+label, so a `Duration` param's first occurrence (which pushes no `Bucket`) still lets the next occurrence compute elapsed time.
LastMetricEvent {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
/// Marker for the end of the whole range. Nothing sorts after this.
End,
}
src/main.rs
+4 -150
View File
@@ -1,156 +1,10 @@
mod alerts;
mod api; mod api;
mod data_value; mod cli;
mod dto; mod config;
mod consts;
mod engine; mod engine;
mod filter_datetime;
mod filter_parser;
mod index_key;
mod ingest;
mod metric_primitive;
mod profile;
mod promql;
mod settings;
mod timestamp;
use std::path::PathBuf;
use std::sync::{Arc, LazyLock, Mutex};
use clap::Parser;
use tokio::net::TcpListener;
use tokio::sync::mpsc;
use profile::ProfileStore;
use settings::Settings;
/// The resolved default settings path, rendered into `--help` so `--settings-path`'s help text
/// names a real, concrete location instead of only describing where to look for one.
static SETTINGS_PATH_HELP: LazyLock<String> = LazyLock::new(|| {
format!(
"Path to the settings file. Defaults to {}.",
settings::default_settings_path()
.map(|path| path.display().to_string())
.unwrap_or_else(|| "a per-OS config location".to_string())
)
});
/// Observability dashboard from terminal output.
#[derive(Parser, Debug)]
struct Args {
/// Port to serve the API and webapp on. Defaults to 0, letting the OS assign
/// an ephemeral free port.
#[arg(long, default_value_t = 0)]
port: u16,
#[arg(long, help = SETTINGS_PATH_HELP.as_str())]
settings_path: Option<PathBuf>,
/// File to tail for log lines, instead of reading from stdin. Behaves like `tail -f`.
#[arg(long)]
file: Option<PathBuf>,
/// URL of a Prometheus/OpenMetrics-compatible endpoint to poll for metrics, in addition to log ingestion.
#[arg(long)]
metrics_url: Option<String>,
}
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
let args = Args::parse(); cli::run().await;
let settings_path = args.settings_path.clone().unwrap_or_else(|| {
settings::default_settings_path()
.expect("could not determine a default settings path; pass --settings-path")
});
let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
let (engine_tx, engine_rx) = mpsc::channel(1024);
tokio::spawn(engine::run(engine_rx, settings.clone()));
{
let max_logs = settings.lock().unwrap().max_logs;
let _ = engine_tx.send(engine::EngineMessage::SetMaxLogs(max_logs)).await;
}
// Reload the active profile's declarations before ingest starts: there's no way to
// reprocess early lines later, so declaring after ingest starts could miss lines that raced ahead.
{
let (active_profile_name, profiles_dir) = {
let mut settings_guard = settings.lock().unwrap();
let profiles_dir = profile::resolve_profiles_dir(&settings_guard, &settings_path);
let mut store = ProfileStore::load(&profile::profiles_store_path(&profiles_dir));
store
.ensure_default(&profile::profiles_store_path(&profiles_dir), &mut settings_guard, &settings_path)
.expect("failed to ensure the Default profile exists");
(settings_guard.profiles.active_profile.clone(), profiles_dir)
};
if let Some(profile_name) = active_profile_name {
let store = ProfileStore::load(&profile::profiles_store_path(&profiles_dir));
match store.get(&profile_name) {
Some(active_profile) => {
profile::load_into_engine(&engine_tx, active_profile).await;
}
None => {
eprintln!(
"warning: active profile '{profile_name}' not found under {}; skipping",
profiles_dir.display()
);
}
}
}
}
match args.file.clone() {
Some(path) => {
tokio::spawn(ingest::run_file(engine_tx.clone(), path));
}
None => {
tokio::spawn(ingest::run(engine_tx.clone()));
}
}
if let Some(metrics_url) = args.metrics_url.clone() {
tokio::spawn(ingest::run_metrics_poll(engine_tx.clone(), metrics_url));
}
let app = api::router(engine_tx, settings, settings_path);
let listener = TcpListener::bind(("0.0.0.0", args.port))
.await
.expect("failed to bind listener");
let bound_port = listener
.local_addr()
.expect("failed to read back bound address")
.port();
let source = match &args.file {
Some(path) => format!("tailing {}", path.display()),
None => "reading logs from stdin".to_string(),
};
eprintln!(
"ottyl serving on http://localhost:{} ({})",
bound_port, source
);
axum::serve(listener, app)
.await
.expect("server error");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn settings_path_defaults_to_none() {
let args = Args::parse_from(["ottyl"]);
assert_eq!(args.settings_path, None);
}
#[test]
fn settings_path_flag_overrides_default() {
let args = Args::parse_from(["ottyl", "--settings-path", "/tmp/custom-settings.json"]);
assert_eq!(
args.settings_path,
Some(PathBuf::from("/tmp/custom-settings.json"))
);
}
} }
src/profile.rs
-1114
View File
File diff suppressed because it is too large Load Diff
tickets/077-log-queries-scan-full-index-instead-of-seeking-by-key-range.md
+72
View File
@@ -0,0 +1,72 @@
# 077 — Log queries scan the full index instead of seeking by key range
## Summary
Every log read — including polling reads that only want logs newer
than `since_sequence` — walks the *entire* retained log range and
filters with a closure, rather than seeking the `BTreeMap` directly to
the relevant key range. This is the read-path optimization the
`left-right`/sort-helper-index work never got around to: iterators
should narrow the underlying key range as far as the caller's bounds
allow, and accept range bounds as a parameter instead of scanning from
`Start`/`MetricBucketSplit` and filtering after the fact.
## Where
- `EngineSnapshot::logs_range` (`src/engine/snapshot/mod.rs:159-168`)
always ranges `IndexKey::Start..IndexKey::LogParamSplit` — i.e. every
ingested log ever retained — with no way for a caller to pass a
narrower bound.
- `EngineSnapshot::logs_since` (`src/engine/snapshot/mod.rs:170-181`)
calls `logs_range()` and then filters by `sequence < before_sequence`
and a timestamp window in a closure — it has all the information
needed to seek instead (`before_sequence` bounds `sequence`
directly), but throws that away and scans linearly regardless of how
small the actual window is.
- The unsorted log-query path in `handle_query_inner`
(`src/engine/snapshot/query.rs:59-65`) does the same thing:
`snapshot.logs_range().filter(|(sequence, timestamp, _)| { since ...
start_ms ... end_ms ... })`. On every poll (frontend passes
`since_sequence` — see `webapp/src/lib/api.ts:263,343`), this walks
every previously-seen log line again just to throw it away in the
filter, before even getting to the new ones.
- `IndexKey::IngestedLog { sequence, timestamp }`
(`src/engine/index_key.rs:10-13`) derives `Ord` field-order-first on
`sequence`, so a lower bound of
`IndexKey::IngestedLog { sequence: since + 1, timestamp: Timestamp::MIN }`
(or equivalent) is a valid, correct seek — sequence is assigned
monotonically at ingest (`src/engine/handle.rs:118-119`), so this
isn't a heuristic, it's exact.
## Scope
- Change `logs_range` to accept a key range (e.g.
`RangeBounds<IndexKey>` or a `(Bound<u64>, Bound<u64>)` over
sequence) instead of hardcoding `Start..LogParamSplit`, so callers
can seek.
- Rework `logs_since` and the `EngineQuery::Logs` handler in
`query.rs:59-65` to build a lower bound from `since_sequence` (and,
where possible, from `start_ms`/note below) and pass it through,
instead of filtering post-hoc from the very start of the index.
- Note: because `IngestedLog` keys sort by `(sequence, timestamp)` and
not `(timestamp, sequence)`, a `start_ms`/`end_ms`-only query (no
`since_sequence`) can't be turned into an exact key-range seek from
this index alone even though ingestion is roughly time-ordered by
sequence — call this out explicitly rather than seeking incorrectly;
it may be acceptable to leave that case as a scan, or to binary-search
for the first sequence at/after `start_ms` since timestamps are
non-decreasing with sequence in practice.
- The sorted branch (`query.rs:82-128`) iterates `sort_helper_index`
directly rather than `logs_range`, so it's less affected, but still
does a per-candidate `index.get()` (`query.rs:106-107`) with no early
bound — check whether the helper index itself can be range-bounded
once `since_sequence`/`start_ms` are known.
- Add/extend a test asserting that a `since_sequence` poll on a large
log store does not touch (or does not compare against) log entries
at or before `since_sequence` — a range-based seek should make this
observably true, not just faster.
## Depends on
None. Related to ticket 060/061 (sort helper indexes) — same
underlying principle (avoid full scans by indexing/seeking).
tickets/078-metric-comparison-filter-scans-all-buckets-for-every-query.md
+56
View File
@@ -0,0 +1,56 @@
# 078 — Metric-comparison filters scan every metric's buckets on every query
## Summary
Same underlying issue as ticket 077 (iterators should narrow the key
range as far as possible, and take a range as a parameter), applied to
metrics: a metric-comparison filter (e.g. `latency_ms[value]<200` in
the filter bar) rebuilds its matching-sequence set from scratch on
every single query by scanning *all* buckets for *all* metrics, then
discarding everything that doesn't match the requested metric name —
instead of seeking directly to that metric's key range.
## Where
- `EngineSnapshot::bucket_values`
(`src/engine/snapshot/metrics.rs:295-312`) ranges
`IndexKey::MetricBucketSplit..IndexKey::End` — every bucket for every
declared metric — and filters by `name == metric_name` inside the
`filter_map` closure. `IndexKey::Bucket { metric_name, labels }`
(`src/engine/index_key.rs:23-26`) derives `Ord` on `metric_name`
first, so this is seekable: a lower bound of
`IndexKey::Bucket { metric_name: metric_name.into(), labels:
BTreeMap::new() }` and an upper bound one metric name past it would
narrow the range to just that metric's buckets.
- The compare-filter compiler,
`compile_metric_comparison` (`src/engine/snapshot/filter.rs:83-104`),
calls `self.bucket_values(metric, None)` and builds a fresh
`HashSet<u64>` of matching sequences on every call — this runs once
per filter compile, and filters are compiled fresh per query
(`compile_filter_query`, called from
`src/engine/snapshot/query.rs:46`), so a dashboard or search bar
polling with a metric-comparison filter re-scans all bucket values
every poll interval. Unlike the log sort-helper indexes
(`src/engine/snapshot/sort.rs`), there is no incremental cache here
keyed by `(metric, param, op, value)`.
## Scope
- Narrow `bucket_values` to accept/derive a key range for the
requested `metric_name` instead of ranging
`MetricBucketSplit..End` and filtering by name in the closure. This
alone removes the "every other metric's buckets" part of the scan
regardless of caching.
- Separately (bigger win, more invasive — may warrant its own
follow-up ticket): consider whether metric-comparison results can be
incrementally maintained similarly to `sort_helper_indexes`
(`src/engine/snapshot/sort.rs`) rather than recomputed per query, if
profiling shows compare-filtered dashboard polling is a hot path.
Keep this ticket scoped to the range-narrowing fix; note the caching
idea here so it isn't lost, but split it out if it turns out to need
design work (cache invalidation on new bucket values, per unique
filter expression).
## Depends on
None. Same principle as ticket 077.
tickets/079-free-text-regex-filter-has-no-index-full-scan-per-query.md
+56
View File
@@ -0,0 +1,56 @@
# 079 — Free-text regex filters have no index; full scan + regex eval per query
## Summary
A free-text search-bar filter (a bare regex, not a
`param[value]op`/`ottyl_seq`/`ottyl_ts` comparison) is evaluated
line-by-line against every retained log on every query — there's no
narrower key range to seek into here since the match isn't tied to a
declared param or timestamp, so this is the one case in the read path
where "narrow the key range" doesn't directly apply and an actual
secondary index would be needed instead.
## Where
- `CompiledFilter::Regex` (compiled in
`src/engine/snapshot/filter.rs`, evaluated via `.eval(...)` per log
line) is checked against the full text of every candidate log line
returned by `logs_range()`/the sort-helper iteration in
`handle_query_inner` (`src/engine/snapshot/query.rs:59-70` and
`111-116`) — i.e. once ticket 077 narrows the *sequence*/time range,
a regex-text filter still forces a full per-line regex match over
everything left in that range, and if there's no `since_sequence`
bound at all (e.g. a fresh search over the whole retained log
window), this is a full linear scan with a regex eval per line.
- There is no cached/precomputed text index (trigram, token, or
otherwise) — every query recompiles nothing (the regex itself is
cached fine, see param/metric regex caches in
`src/engine/snapshot/mod.rs:26-34`), but re-*evaluates* the regex
against the full text of every line in range, every time.
## Scope
- This is the most invasive of the read-path tickets — flag as such.
Do not implement speculatively; first confirm via profiling
(`samply record` against a realistic log volume with an active
free-text filter) that this is actually the dominant cost before
building an index, since regex eval over a bounded/paginated result
set may already be fast enough once ticket 077's range-narrowing
lands.
- If profiling confirms it's worth it: investigate a coarse
pre-filter (e.g. a trigram index or per-line token set) built
incrementally at ingest time (alongside
`update_sort_helper_indexes_on_ingest`,
`src/engine/handle.rs:151`) that can cheaply reject lines that
can't possibly match a given regex's required literal substrings,
before paying for the actual regex match — not a replacement for
the regex eval, a way to skip most of it.
- Keep the index bounded in size / evictable in step with
`evict_excess_logs` (`src/engine/handle.rs:153`) so it doesn't grow
unbounded relative to `max logs in memory`.
## Depends on
Should land after ticket 077 (range narrowing) so profiling reflects
the scan cost that's actually left, not the one about to be fixed
anyway.
tickets/080-json-field-filter-reparses-json-per-line-per-query.md
+52
View File
@@ -0,0 +1,52 @@
# 080 — JSON field filters re-parse every candidate line's JSON on every query
## Summary
A filter comparing a JSON field (e.g. a log line shaped as JSON,
filtered by a field inside it) re-parses the full JSON body of every
candidate log line from scratch on every query that uses such a
filter — there's no memoization of the parse, even though the same log
line's JSON gets re-parsed identically on every subsequent poll until
the line ages out.
## Where
- `eval_json_comparison` (`src/engine/snapshot/filter.rs:177-185`)
calls `serde_json::from_str::<serde_json::Value>(text)` on the raw
log text every time it's invoked, and it's invoked once per
candidate line per query (called from the per-line filter
evaluation path used in `query.rs:59-70`/`111-116` via
`CompiledFilter::eval`).
- Contrast with `log_entry_params` (used a few lines below in
`query.rs:79`/`121`), which at least only runs against lines that
survive the filter, not before — the JSON parse here runs as part of
filtering itself, so it can't be skipped even for lines that get
discarded downstream for other reasons if this filter is combined
with others.
## Scope
- Lowest-effort option: nothing to cache at the snapshot level today
since `DataValue::LogLiteral` only stores raw text
(`src/engine/snapshot/mod.rs` / `data_value.rs`) — re-parsing per
query is currently unavoidable without changing storage. Confirm
this is worth doing before storing pre-parsed JSON: check whether
most ingested lines are actually JSON-shaped in typical usage, since
adding a parsed-JSON cache to every ingested line penalizes the
(likely more common) non-JSON case.
- If justified, consider parsing once at ingest time only for lines
that look JSON-shaped (reuse the cheap detection already implied by
`eval_json_comparison`'s `Ok(...) else return false` short-circuit)
and storing the parsed `serde_json::Value` (or just the needed field
extractions) alongside `LogLiteral`, invalidated/dropped the same
way the line itself is evicted (`evict_excess_logs`,
`src/engine/handle.rs:153`).
- Alternative, cheaper fix: memoize the parse per-query-invocation
rather than per-line-per-filter-eval, i.e. if a single query
applies the JSON filter against a line more than once (it currently
doesn't appear to, but double-check `CompiledFilter` composition —
AND/OR combinators) avoid re-parsing within that single pass.
## Depends on
None.
tickets/081-promql-dashboard-queries-rebuild-full-series-from-scratch-per-poll.md
+52
View File
@@ -0,0 +1,52 @@
# 081 — PromQL dashboard queries rebuild full label→points maps from scratch every poll
## Summary
A dashboard panel backed by a PromQL-style query re-evaluates its
entire series (all labels, all points) from scratch on every polling
interval, with no reuse of the previous evaluation's work and no
incremental narrowing by the time range the panel actually needs —
same "narrow the range, don't rescan everything" principle as the
other read-path tickets, applied to the dashboard/metrics query path
this time.
## Where
- `evaluate_promql` and its helpers
(`src/engine/snapshot/query.rs:412-548`) build label→points maps for
the query's underlying metric(s) fresh on every call — there is no
reuse across repeated calls with the same query string and an
overlapping time window, which is exactly the pattern dashboard
polling produces (`webapp/src/lib/polling.ts` re-requests the same
panel query on an interval).
- This sits on top of `bucket_values` (ticket 078), so once that's
narrowed to a metric's own key range, `evaluate_promql` still
recomputes the full aggregation/label-grouping logic over whatever
`bucket_values` returns, every single call, even for the portion of
the time range it already computed on the previous poll.
## Scope
- Start by checking whether panels pass a time range/step down to
`evaluate_promql` at all — if they do, apply the same key-range
narrowing as ticket 077/078 first (bound the underlying
`bucket_values` range by the panel's requested window) before
considering caching, since that alone may remove most of the
redundant work.
- If polling the *same* query with an advancing time window is common
(sliding dashboard windows), investigate caching prior evaluation
results keyed by `(query string, label set)` and only computing the
delta for the newly-advanced portion of the window — this is more
involved (needs a cache invalidation story tied to new bucket
ingestion, similar to `sort_helper_indexes`'
`purge_stale_sort_helper_indexes`,
`src/engine/snapshot/sort.rs`) — do not build this speculatively
without first confirming via profiling that repeated PromQL
evaluation (not `bucket_values` scanning, addressed by ticket 078)
is the actual bottleneck for dashboards with multiple panels.
## Depends on
Should land after ticket 078 (metric bucket range narrowing), since
that changes what `evaluate_promql` is scanning underneath and may
reduce the size of this problem before any caching work is designed.
webapp/index.html
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8" /> <meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" /> <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>webapp</title> <title>ottyl</title>
</head> </head>
<body> <body>
<div id="root"></div> <div id="root"></div>
webapp/public/favicon.svg
+26 -1
View File
File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 9.3 KiB

After

Width:  |  Height:  |  Size: 669 B

webapp/src/App.tsx
+105 -7
View File
@@ -6,6 +6,9 @@ import {
TIME_RANGE_PRESETS, TIME_RANGE_PRESETS,
computeBucketMs, computeBucketMs,
computePresetRange, computePresetRange,
loadLogSortSettings,
saveLogSortSettings,
type LogSortSettings,
type TimeRange, type TimeRange,
type TimeRangePresetId, type TimeRangePresetId,
} from './components/LogTimeline' } from './components/LogTimeline'
@@ -66,6 +69,23 @@ function saveLogViewerHeight(px: number) {
} }
} }
// Ticket 074: a sort helper index's one-time full-scan build (`build_sort_helper_index`) runs
// inline in the engine actor, so a `sort_by` poll can occasionally take much longer than the usual
// sub-poll-interval round trip — without this, that stall looks exactly like "sorting is broken"
// rather than "still indexing". Only flips `setBuilding(true)` if the fetch is still in flight after
// `thresholdMs`, so an ordinary fast poll (helper index already built/incrementally maintained)
// never flashes the indicator.
const SORT_BUILDING_INDICATOR_THRESHOLD_MS = 400
function trackSlowSortedFetch<T>(sortBy: string | null, setBuilding: (building: boolean) => void, promise: Promise<T>): Promise<T> {
if (!sortBy) return promise
const timer = setTimeout(() => setBuilding(true), SORT_BUILDING_INDICATOR_THRESHOLD_MS)
return promise.finally(() => {
clearTimeout(timer)
setBuilding(false)
})
}
/// Thin drag handle between the log viewer and the sections below it. Reports each pointer-move /// Thin drag handle between the log viewer and the sections below it. Reports each pointer-move
/// delta via `onResize` (letting the caller clamp/apply it) and fires `onResizeEnd` on release so /// delta via `onResize` (letting the caller clamp/apply it) and fires `onResizeEnd` on release so
/// the final size can be persisted once, not on every pixel of movement. /// the final size can be persisted once, not on every pixel of movement.
@@ -251,6 +271,28 @@ function StandaloneRanged({
) )
} }
/// Ticket 058: only replaces `params` state when the declaration list actually changed, so a param
/// list refetched on every poll tick doesn't force `Params.tsx` to re-render when nothing changed.
function updateParamsIfChanged(
setParams: (updater: (current: ParamDeclaration[]) => ParamDeclaration[]) => void,
next: ParamDeclaration[],
) {
setParams((current) => (JSON.stringify(current) === JSON.stringify(next) ? current : next))
}
/// Ticket 066: text shown under the filter bar when the user hasn't typed a manual filter, so an active preset/sort order is still visible somewhere near the filter input instead of leaving it looking empty/stale.
function describeActiveState(activePreset: TimeRangePresetId | null, sortSettings: LogSortSettings): string | null {
const parts: string[] = []
if (activePreset && activePreset !== 'all') {
const preset = TIME_RANGE_PRESETS.find((p) => p.id === activePreset)
parts.push(`last ${preset?.label ?? activePreset}`)
}
if (sortSettings.sortBy) {
parts.push(`sorted by ${sortSettings.sortBy}`)
}
return parts.length > 0 ? parts.join(', ') : null
}
/// The log viewer has its own `LogTimeline`, so it needs its own copy of App's log/timeline polling state instead of a carried-over range. /// The log viewer has its own `LogTimeline`, so it needs its own copy of App's log/timeline polling state instead of a carried-over range.
function StandaloneLog() { function StandaloneLog() {
const [logs, setLogs] = useState<LogEntry[]>([]) const [logs, setLogs] = useState<LogEntry[]>([])
@@ -260,6 +302,13 @@ function StandaloneLog() {
const [activePreset, setActivePreset] = useState<TimeRangePresetId | null>('all') const [activePreset, setActivePreset] = useState<TimeRangePresetId | null>('all')
const [filter, setFilter] = useState('') const [filter, setFilter] = useState('')
const [filterError, setFilterError] = useState<string | null>(null) const [filterError, setFilterError] = useState<string | null>(null)
const [sortSettings, setSortSettings] = useState<LogSortSettings>(loadLogSortSettings)
const [sortIndexBuilding, setSortIndexBuilding] = useState(false)
function handleSortSettingsChange(next: LogSortSettings) {
setSortSettings(next)
saveLogSortSettings(next)
}
const rangeRef = useRef(range) const rangeRef = useRef(range)
useEffect(() => { useEffect(() => {
@@ -302,7 +351,7 @@ function StandaloneLog() {
) )
} }
try { try {
const [nextBuckets, nextLogs] = await Promise.all([ const [nextBuckets, nextLogs, nextParams] = await Promise.all([
fetchTimeline({ fetchTimeline({
startMs: effectiveRange?.startMs, startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs, endMs: effectiveRange?.endMs,
@@ -310,11 +359,25 @@ function StandaloneLog() {
bucketMs: effectiveRange ? computeBucketMs(effectiveRange.endMs - effectiveRange.startMs) : undefined, bucketMs: effectiveRange ? computeBucketMs(effectiveRange.endMs - effectiveRange.startMs) : undefined,
filter: filter || undefined, filter: filter || undefined,
}), }),
fetchLogs({ startMs: effectiveRange?.startMs, endMs: effectiveRange?.endMs, limit: 1000, filter: filter || undefined }), trackSlowSortedFetch(
sortSettings.sortBy,
setSortIndexBuilding,
fetchLogs({
startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs,
limit: 1000,
filter: filter || undefined,
sortBy: sortSettings.sortBy ?? undefined,
sortMissingFallback: sortSettings.sortBy ? sortSettings.sortMissingFallback : undefined,
}),
),
// Ticket 058: params can get auto-declared from freshly ingested lines mid-run, not just from a user's own edit.
fetchParamDeclarations(),
]) ])
if (!cancelled) { if (!cancelled) {
setBuckets(nextBuckets) setBuckets(nextBuckets)
setLogs(nextLogs) setLogs(nextLogs)
updateParamsIfChanged(setParams, nextParams)
setFilterError(null) setFilterError(null)
} }
} catch (err) { } catch (err) {
@@ -331,12 +394,17 @@ function StandaloneLog() {
clearInterval(id) clearInterval(id)
} }
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
}, [filter, activePreset]) }, [filter, activePreset, sortSettings])
return ( return (
<StandalonePage> <StandalonePage>
<div className="flex h-full min-h-0 flex-col"> <div className="flex h-full min-h-0 flex-col">
<FilterBar value={filter} onChange={setFilter} error={filterError} /> <FilterBar
value={filter}
onChange={setFilter}
error={filterError}
activeSummary={describeActiveState(activePreset, sortSettings)}
/>
<div className="flex min-h-0 flex-1 flex-col"> <div className="flex min-h-0 flex-1 flex-col">
<LogTimeline <LogTimeline
buckets={buckets} buckets={buckets}
@@ -344,6 +412,10 @@ function StandaloneLog() {
onSelectRange={handleManualRangeSelect} onSelectRange={handleManualRangeSelect}
activePreset={activePreset} activePreset={activePreset}
onSelectPreset={selectPreset} onSelectPreset={selectPreset}
paramNames={params.map((p) => p.name)}
sortSettings={sortSettings}
onSortSettingsChange={handleSortSettingsChange}
sortIndexBuilding={sortIndexBuilding}
/> />
<LogPreview logs={logs} params={params} /> <LogPreview logs={logs} params={params} />
</div> </div>
@@ -373,6 +445,13 @@ function FullApp() {
const [filterError, setFilterError] = useState<string | null>(null) const [filterError, setFilterError] = useState<string | null>(null)
const [logViewerCollapsed, setLogViewerCollapsed] = useSectionCollapsed('log-viewer') const [logViewerCollapsed, setLogViewerCollapsed] = useSectionCollapsed('log-viewer')
const [logViewerHeight, setLogViewerHeight] = useState(() => loadLogViewerHeight()) const [logViewerHeight, setLogViewerHeight] = useState(() => loadLogViewerHeight())
const [sortSettings, setSortSettings] = useState<LogSortSettings>(loadLogSortSettings)
const [sortIndexBuilding, setSortIndexBuilding] = useState(false)
function handleSortSettingsChange(next: LogSortSettings) {
setSortSettings(next)
saveLogSortSettings(next)
}
function handleLogViewerResize(deltaY: number) { function handleLogViewerResize(deltaY: number) {
setLogViewerHeight((current) => clampLogViewerHeight(current + deltaY)) setLogViewerHeight((current) => clampLogViewerHeight(current + deltaY))
@@ -433,7 +512,7 @@ function FullApp() {
) )
} }
try { try {
const [nextStats, nextBuckets, nextLogs] = await Promise.all([ const [nextStats, nextBuckets, nextLogs, nextParams] = await Promise.all([
fetchStats(), fetchStats(),
fetchTimeline({ fetchTimeline({
startMs: effectiveRange?.startMs, startMs: effectiveRange?.startMs,
@@ -442,12 +521,26 @@ function FullApp() {
bucketMs: effectiveRange ? computeBucketMs(effectiveRange.endMs - effectiveRange.startMs) : undefined, bucketMs: effectiveRange ? computeBucketMs(effectiveRange.endMs - effectiveRange.startMs) : undefined,
filter: filter || undefined, filter: filter || undefined,
}), }),
fetchLogs({ startMs: effectiveRange?.startMs, endMs: effectiveRange?.endMs, limit: 1000, filter: filter || undefined }), trackSlowSortedFetch(
sortSettings.sortBy,
setSortIndexBuilding,
fetchLogs({
startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs,
limit: 1000,
filter: filter || undefined,
sortBy: sortSettings.sortBy ?? undefined,
sortMissingFallback: sortSettings.sortBy ? sortSettings.sortMissingFallback : undefined,
}),
),
// Ticket 058: params can get auto-declared from freshly ingested lines mid-run, not just from a user's own edit.
fetchParamDeclarations(),
]) ])
if (!cancelled) { if (!cancelled) {
setStats(nextStats) setStats(nextStats)
setBuckets(nextBuckets) setBuckets(nextBuckets)
setLogs(nextLogs) setLogs(nextLogs)
updateParamsIfChanged(setParams, nextParams)
setFilterError(null) setFilterError(null)
} }
} catch (err) { } catch (err) {
@@ -465,7 +558,7 @@ function FullApp() {
} }
// Intentionally excludes `range` to avoid restarting this interval on every rolling-preset tick. // Intentionally excludes `range` to avoid restarting this interval on every rolling-preset tick.
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
}, [filter, activePreset]) }, [filter, activePreset, sortSettings])
return ( return (
<DocsNavProvider> <DocsNavProvider>
@@ -476,6 +569,7 @@ function FullApp() {
value={filter} value={filter}
onChange={setFilter} onChange={setFilter}
error={filterError} error={filterError}
activeSummary={describeActiveState(activePreset, sortSettings)}
collapseToggle={ collapseToggle={
<> <>
<CollapseToggle collapsed={logViewerCollapsed} onToggle={() => setLogViewerCollapsed(!logViewerCollapsed)} /> <CollapseToggle collapsed={logViewerCollapsed} onToggle={() => setLogViewerCollapsed(!logViewerCollapsed)} />
@@ -497,6 +591,10 @@ function FullApp() {
onSelectRange={handleManualRangeSelect} onSelectRange={handleManualRangeSelect}
activePreset={activePreset} activePreset={activePreset}
onSelectPreset={selectPreset} onSelectPreset={selectPreset}
paramNames={params.map((p) => p.name)}
sortSettings={sortSettings}
onSortSettingsChange={handleSortSettingsChange}
sortIndexBuilding={sortIndexBuilding}
/> />
<LogPreview logs={logs} params={params} /> <LogPreview logs={logs} params={params} />
</div> </div>
webapp/src/assets/ottyl-icon.svg
+26
View File
@@ -0,0 +1,26 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
<defs>
<style>
.cls-1 {
fill: #fff;
stroke-width: 9px;
}
.cls-1, .cls-2 {
stroke: #000;
stroke-miterlimit: 10;
}
.cls-2 {
fill: none;
stroke-width: 8px;
stroke-linecap: round;
stroke-linejoin: round;
}
</style>
</defs>
<rect class="cls-1" x="10" y="10" width="80" height="80" rx="18.79" ry="18.79"/>
<polyline class="cls-2" points="25,33 42,50 25,67"/>
<line class="cls-2" x1="46" y1="66" x2="75" y2="37"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 669 B

webapp/src/components/Dashboards.tsx
+21 -3
View File
@@ -96,7 +96,7 @@ function PanelPreview({ panel, range }: { panel: Panel; range: TimeRange | null
if (loading && series.length === 0) { if (loading && series.length === 0) {
return <div className="flex h-48 items-center justify-center text-xs text-text-faint">Loading…</div> return <div className="flex h-48 items-center justify-center text-xs text-text-faint">Loading…</div>
} }
return <PanelChart series={series} chartType={panel.chart_type} /> return <PanelChart series={series} chartType={panel.chart_type} range={range} />
} }
function AutosaveStatus({ status, error }: { status: SaveStatus; error: string | null }) { function AutosaveStatus({ status, error }: { status: SaveStatus; error: string | null }) {
@@ -153,6 +153,9 @@ export function Dashboards({ range, activePreset }: { range: TimeRange | null; a
const [saveError, setSaveError] = useState<string | null>(null) const [saveError, setSaveError] = useState<string | null>(null)
// Last-persisted shape for the current draft, so the autosave effect can skip when nothing actually changed. // Last-persisted shape for the current draft, so the autosave effect can skip when nothing actually changed.
const lastSavedRef = useRef<string | null>(null) const lastSavedRef = useRef<string | null>(null)
// Ticket 064: latest `draft`, read inside the autosave timeout's async callback so a save response can tell
// whether the user kept typing after the request went out, instead of unconditionally overwriting `draft`.
const draftRef = useRef<Dashboard | null>(draft)
const [collapsed, setCollapsed] = useSectionCollapsed('dashboards') const [collapsed, setCollapsed] = useSectionCollapsed('dashboards')
async function loadDashboards(profileName: string, selectAfterLoad?: string) { async function loadDashboards(profileName: string, selectAfterLoad?: string) {
@@ -181,11 +184,21 @@ export function Dashboards({ range, activePreset }: { range: TimeRange | null; a
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
}, []) }, [])
useEffect(() => {
draftRef.current = draft
}, [draft])
// Deliberately keyed only on `selectedId`, not `dashboards` — this effect's job is to load a *newly selected*
// dashboard's content into `draft`. Keying it on `dashboards` too meant every autosave's own `setDashboards`
// call (below) re-ran this effect and clobbered whatever the user had typed since the save was kicked off
// (ticket 064). Reading `dashboards` from the closure here still sees the just-loaded array, since
// `setSelectedId`/`setDashboards` from `loadDashboards` land in the same render.
useEffect(() => { useEffect(() => {
const found = dashboards.find((d) => d.id === selectedId) ?? null const found = dashboards.find((d) => d.id === selectedId) ?? null
setDraft(found) setDraft(found)
lastSavedRef.current = found ? JSON.stringify(dashboardSaveShape(found)) : null lastSavedRef.current = found ? JSON.stringify(dashboardSaveShape(found)) : null
}, [selectedId, dashboards]) // eslint-disable-next-line react-hooks/exhaustive-deps
}, [selectedId])
// Skips entirely once the draft matches what's already persisted — covers both re-selecting an unchanged dashboard and this effect's own post-save re-render. // Skips entirely once the draft matches what's already persisted — covers both re-selecting an unchanged dashboard and this effect's own post-save re-render.
useEffect(() => { useEffect(() => {
@@ -198,9 +211,14 @@ export function Dashboards({ range, activePreset }: { range: TimeRange | null; a
setSaveStatus('saving') setSaveStatus('saving')
try { try {
const saved = await saveDashboard(activeProfile, dashboardId, dashboardSaveShape(draft)) const saved = await saveDashboard(activeProfile, dashboardId, dashboardSaveShape(draft))
lastSavedRef.current = JSON.stringify(dashboardSaveShape(saved)) lastSavedRef.current = snapshot
setDashboards((prev) => prev.map((d) => (d.id === saved.id ? saved : d))) setDashboards((prev) => prev.map((d) => (d.id === saved.id ? saved : d)))
// Only reconcile `saved` into `draft` if nothing newer was typed while the request was in flight —
// otherwise this stale response would overwrite those in-progress keystrokes (ticket 064).
const current = draftRef.current
if (current && current.id === dashboardId && JSON.stringify(dashboardSaveShape(current)) === snapshot) {
setDraft(saved) setDraft(saved)
}
setSaveStatus('saved') setSaveStatus('saved')
setSaveError(null) setSaveError(null)
} catch (err) { } catch (err) {
webapp/src/components/Docs.tsx
+145 -33
View File
@@ -50,9 +50,10 @@ const SECTIONS: DocSection[] = [
</ul> </ul>
<p className="font-semibold text-text">Settings</p> <p className="font-semibold text-text">Settings</p>
<p> <p>
One <em>settings</em> file per machine holds the colour scheme, imported/linked CSS, the ntfy sink, and One <em>settings</em> file per machine holds the active style (a built-in colour scheme or an
which profile is active. Settings only tracks profile pointers — the profiles themselves (params, metrics, imported/linked stylesheet — see below), the ntfy sink, and which profile is active. Settings only tracks
dashboards) are stored separately, next to the settings file by default. profile pointers — the profiles themselves (params, metrics, dashboards) are stored separately, next to the
settings file by default.
</p> </p>
<p className="font-semibold text-text">Profiles</p> <p className="font-semibold text-text">Profiles</p>
<p> <p>
@@ -77,32 +78,62 @@ const SECTIONS: DocSection[] = [
id: 'filtering', id: 'filtering',
title: 'Filtering', title: 'Filtering',
body: ( body: (
<div className="flex flex-col gap-2"> <div className="flex flex-col gap-3">
<p> <p>
The filter bar combines regex search over raw log text with comparisons against declared metrics, joined The filter bar combines regex search over raw log text with comparisons against declared metrics, joined
by <Code>and</Code>/<Code>or</Code>. Examples: by <Code>and</Code>/<Code>or</Code>. The server evaluates the filter string; a parse error shows as inline
text under the filter bar.
</p>
<div>
<p className="font-semibold text-text">Regex match</p>
<ul className="list-disc pl-5">
<li>
<Code>{'regex("error")'}</Code> — lines containing "error". See the <Code>@</Code> icon next to the
filter bar for the exact regex syntax supported (Rust's <Code>regex</Code> crate — no lookaround or
backreferences).
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Metric/param comparisons</p>
<ul className="list-disc pl-5">
<li>
<Code>{'requests[duration]>100'}</Code> — the <Code>requests</Code> metric's captured{' '}
<Code>duration</Code> param over 100. <Code>[param]</Code> comparisons only work against a{' '}
<em>declared metric's</em> labels, not a bare param declared on its own, and values are unquoted (no
surrounding <Code>"..."</Code>).
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Reserved names (always filterable)</p>
<p>
<Code>ottyl_seq</Code>, <Code>ottyl_ts</Code>, and <Code>ottyl_fmt</Code> are present on every log line
without declaring anything, and also appear (flagged <Code>reserved</Code>) in{' '}
<Code>GET /api/params</Code> for discoverability, though there's no UI to edit or remove them.
</p> </p>
<ul className="list-disc pl-5"> <ul className="list-disc pl-5">
<li> <li>
<Code>{'regex("error")'}</Code> — lines containing "error". <Code>{'ottyl_seq<5'}</Code> — ingest sequence number under 5.
</li> </li>
<li> <li>
<Code>{'regex("error") and sequence<5'}</Code> — lines containing "error" with sequence under 5. <Code>{'ottyl_ts>-5m'}</Code> — lines from the last 5 minutes (see Date/time formats below).
</li> </li>
<li> <li>
<Code>{'requests[duration]>100'}</Code> — the <Code>requests</Code> metric's captured{' '} <Code>{'ottyl_fmt==json'}</Code> — the line's auto-detected text format:{' '}
<Code>duration</Code> param over 100. <Code>log_stream</Code> (macOS <Code>log stream</Code> output — its columns are also auto-declared as
</li> their own named params, e.g. <Code>PID</Code>/<Code>Thread</Code>, from that line's own header), <Code>json</Code>,{' '}
<li> <Code>logfmt</Code>, or <Code>none</Code>.
<Code>{'timestamp>-5m'}</Code> — lines from the last 5 minutes.
</li> </li>
</ul> </ul>
<p> </div>
<Code>sequence</Code> and <Code>timestamp</Code> are reserved metric names present on every log line, so
they're always filterable without declaring anything. <div>
</p>
<p className="font-semibold text-text">Date/time formats</p> <p className="font-semibold text-text">Date/time formats</p>
<p>Timestamp comparisons accept three forms:</p> <p>Timestamp comparisons (e.g. against <Code>ottyl_ts</Code>) accept three forms:</p>
<ul className="list-disc pl-5"> <ul className="list-disc pl-5">
<li> <li>
relative offsets from now — <Code>5h</Code> (5 hours from now), <Code>-3m</Code> (3 minutes ago) relative offsets from now — <Code>5h</Code> (5 hours from now), <Code>-3m</Code> (3 minutes ago)
@@ -114,29 +145,100 @@ const SECTIONS: DocSection[] = [
full dates — <Code>2026-01-15</Code>, <Code>2026-01-15T09:30:00Z</Code> full dates — <Code>2026-01-15</Code>, <Code>2026-01-15T09:30:00Z</Code>
</li> </li>
</ul> </ul>
<p> </div>
The filter string is evaluated on the server. A parse error shows as inline text under the filter bar.
</p> <div>
<p className="font-semibold text-text">Combining clauses</p>
<ul className="list-disc pl-5">
<li>
<Code>{'regex("error") and ottyl_seq<5'}</Code> — lines containing "error" with sequence under 5.
</li>
<li>
<Code>{'requests[path]==/checkout and regex("5[0-9]{2}")'}</Code> — 5xx responses on one path, given a{' '}
<Code>requests</Code> metric with a <Code>path</Code> label param.
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Worked examples</p> <p className="font-semibold text-text">Worked examples</p>
<ul className="list-disc pl-5"> <ul className="list-disc pl-5">
<li> <li>
Only lines mentioning "timeout" in the last 15 minutes:{' '} Only lines mentioning "timeout" in the last 15 minutes:{' '}
<Code>{'regex("timeout") and timestamp>-15m'}</Code> <Code>{'regex("timeout") and ottyl_ts>-15m'}</Code>
</li> </li>
<li> <li>
Requests slower than 200ms, once a <Code>requests</Code> metric has a <Code>value</Code>-role{' '} Requests slower than 200ms, once a <Code>requests</Code> metric has a <Code>value</Code>-role{' '}
<Code>duration</Code> param declared: <Code>{'requests[duration]>200'}</Code> <Code>duration</Code> param declared: <Code>{'requests[duration]>200'}</Code>
</li> </li>
<li> <li>
5xx responses on one path, given a <Code>requests</Code> metric with a <Code>path</Code> label param — Everything before noon on a specific date: <Code>{'ottyl_ts<2024-06-15T12:00:00Z'}</Code>
note that <Code>[param]</Code> comparisons only work against a <em>declared metric's</em> labels, not a
bare param declared on its own, and values are unquoted (no surrounding <Code>"..."</Code>):{' '}
<Code>{'requests[path]==/checkout and regex("5[0-9]{2}")'}</Code>
</li>
<li>
Everything before noon on a specific date: <Code>{'timestamp<2024-06-15T12:00:00Z'}</Code>
</li> </li>
</ul> </ul>
<p className="mt-1 text-text-subtle">
See also <em>Dashboards</em> below for using <Code>ottyl_seq</Code>/<Code>ottyl_ts</Code> in a panel
query.
</p>
</div>
</div>
),
},
{
id: 'auto-formats',
title: 'Auto-detected formats',
body: (
<div className="flex flex-col gap-2">
<p>
Every ingested line is classified into one of four formats, exposed as the reserved{' '}
<Code>ottyl_fmt</Code> param (see <em>Filtering</em> above). The format also decides what — if anything —
gets auto-declared as a param without you doing anything:
</p>
<ul className="list-disc pl-5">
<li>
<span className="font-semibold text-text">
<Code>log_stream</Code>
</span>{' '}
— macOS <Code>{'log stream'}</Code> output's fixed-column shape (timestamp, thread, type, activity id,
pid, ttl, then <Code>process: message</Code>). The first such line seen auto-declares{' '}
<Code>log_type</Code>, <Code>process</Code>, and <Code>pid</Code> as params, parsed straight out of those
columns.
</li>
<li>
<span className="font-semibold text-text">
<Code>json</Code>
</span>{' '}
— the line parses as a JSON object (a bare array or scalar doesn't count). Every leaf scalar field
(string/number/bool) auto-declares a param named after its dot-separated path, extracted by JSON field
path rather than regex — a nested field like <Code>{'{"request":{"path":"/x"}}'}</Code> becomes a param
named <Code>request.path</Code>. Nested objects are walked into automatically; arrays and <Code>null</Code>{' '}
fields are skipped since there's no single scalar value to declare a param against.
</li>
<li>
<span className="font-semibold text-text">
<Code>logfmt</Code>
</span>{' '}
— the line contains at least one <Code>key=value</Code>/<Code>{'key="value"'}</Code> pair anywhere in its
text (it doesn't need to be logfmt end-to-end — a free-text prefix or suffix around the pairs is fine).
Every distinct key auto-declares a param of the same name, type-inferred as <Code>number</Code> if its
first-seen value parses as one, else <Code>string</Code>.
</li>
<li>
<span className="font-semibold text-text">
<Code>none</Code>
</span>{' '}
— none of the above; plain prose with no structure ottyl can key off of. Nothing is auto-declared, though
the line is still fully searchable via <Code>regex(...)</Code> in the filter bar.
</li>
</ul>
<p>
Auto-declared params are flagged <Code>auto_derived</Code> in <Code>GET /api/params</Code> and show up as
log-table columns immediately, same as a manually declared param — but they can't be edited or removed
individually, and they're never persisted to a profile (so they're rediscovered fresh each run rather than
traveling with a profile like manual declarations do). A manual param declaration for a name always takes
precedence and is never overwritten by auto-detection. To keep one unusual line from flooding the params
list, auto-detection is capped at 64 auto-derived params total and 8 new ones per line — once either cap is
hit, further would-be auto-declarations from new keys/fields are simply skipped.
</p>
</div> </div>
), ),
}, },
@@ -330,6 +432,14 @@ const SECTIONS: DocSection[] = [
Dashboards are saved per profile, so they travel with it when you switch or copy it — an active profile is Dashboards are saved per profile, so they travel with it when you switch or copy it — an active profile is
required to create or save one; pick or create one from the settings menu first. required to create or save one; pick or create one from the settings menu first.
</p> </p>
<p>
The reserved <Code>ottyl_seq</Code>/<Code>ottyl_ts</Code> names (see <em>Filtering</em> above) work in panel
queries the same as any declared metric — e.g. <Code>count(ottyl_seq)</Code> for log volume, or{' '}
<Code>rate(ottyl_ts[1m])</Code> for ingest rate — since every log line carries them without a metric
declaration. <Code>ottyl_fmt</Code> is categorical (<Code>json</Code>/<Code>logfmt</Code>/<Code>none</Code>
), not a number, so it's rejected in a panel query with an explicit error rather than silently charting
nothing — filter on it in the filter bar instead.
</p>
</div> </div>
), ),
}, },
@@ -367,8 +477,9 @@ const SECTIONS: DocSection[] = [
</ul> </ul>
<p> <p>
Query-based conditions (Immediate/Threshold/Ratio) reuse the filter bar's syntax, restricted to{' '} Query-based conditions (Immediate/Threshold/Ratio) reuse the filter bar's syntax, restricted to{' '}
<Code>regex(...)</Code>, <Code>sequence</Code>, <Code>timestamp</Code>, and <Code>and</Code>/<Code>or</Code> <Code>regex(...)</Code>, <Code>ottyl_seq</Code>, <Code>ottyl_ts</Code>, <Code>ottyl_fmt</Code>, and{' '}
. A <Code>metric_name[param]</Code> comparison isn't supported in an alert query. <Code>and</Code>/<Code>or</Code>. A <Code>metric_name[param]</Code> comparison isn't supported in an alert
query.
</p> </p>
<p> <p>
A window accepts a plain number of milliseconds or a humantime-style duration: <Code>30s</Code>,{' '} A window accepts a plain number of milliseconds or a humantime-style duration: <Code>30s</Code>,{' '}
@@ -419,9 +530,10 @@ const SECTIONS: DocSection[] = [
body: ( body: (
<div className="flex flex-col gap-2"> <div className="flex flex-col gap-2">
<p> <p>
The Settings panel controls process-wide state: colour scheme, the ntfy sink alert rules deliver to, the The Settings panel controls process-wide state: the active style — one unified list of built-in colour
max number of logs kept in memory (blank means unlimited), and imported/linked CSS. Changes apply schemes and any imported/linked stylesheets, any of which can be a simple palette or a full layout
immediately and persist to the settings file on disk. There is no save button. redesign — the ntfy sink alert rules deliver to, and the max number of logs kept in memory (blank means
unlimited). Changes apply immediately and persist to the settings file on disk. There is no save button.
</p> </p>
<p className="font-semibold text-text">Worked example</p> <p className="font-semibold text-text">Worked example</p>
<p> <p>
webapp/src/components/FilterBar.tsx
+19 -3
View File
@@ -1,5 +1,5 @@
import type { ReactNode } from 'react' import type { ReactNode } from 'react'
import { useState } from 'react' import { useEffect, useState } from 'react'
import { HelpIcon } from './HelpIcon' import { HelpIcon } from './HelpIcon'
import { RegexHelpIcon } from './RegexHelpIcon' import { RegexHelpIcon } from './RegexHelpIcon'
@@ -13,14 +13,22 @@ export function FilterBar({
onChange, onChange,
error, error,
collapseToggle, collapseToggle,
activeSummary,
}: { }: {
value: string value: string
onChange: (value: string) => void onChange: (value: string) => void
error?: string | null error?: string | null
collapseToggle?: ReactNode collapseToggle?: ReactNode
/** Ticket 066: a read-only textual summary of the active time-range preset / sort order, shown below the input when the user hasn't typed a manual filter — the filter text itself stays the single source of truth for what's actually sent to the server. */
activeSummary?: string | null
}) { }) {
const [draft, setDraft] = useState(value) const [draft, setDraft] = useState(value)
// Resync when `value` changes externally (e.g. a programmatic clear) rather than only reading it once on mount.
useEffect(() => {
setDraft(value)
}, [value])
return ( return (
<form <form
className="flex flex-col gap-1 border-b border-border px-3 py-2" className="flex flex-col gap-1 border-b border-border px-3 py-2"
@@ -30,12 +38,19 @@ export function FilterBar({
}} }}
> >
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<HelpIcon sectionId="filtering" label="Help: Filtering" /> <span className="flex items-center gap-1">
<HelpIcon sectionId="filtering" label="Help: filter syntax" />
<span className="text-xs text-text-subtle">Filter syntax</span>
<span className="mx-0.5 h-4 w-px bg-border-soft" aria-hidden="true" />
<RegexHelpIcon label="Help: regex(...) syntax" /> <RegexHelpIcon label="Help: regex(...) syntax" />
<span className="text-xs text-text-faint" title="Cheat sheet for regex(...) clauses specifically">
regex(...)
</span>
</span>
{collapseToggle} {collapseToggle}
<input <input
className={queryInputClassName} className={queryInputClassName}
placeholder='regex("error") and sequence<5 or timestamp>12:00' placeholder='ottyl_ts>-15m and requests[duration]>200 or regex("error")'
value={draft} value={draft}
onChange={(e) => setDraft(e.target.value)} onChange={(e) => setDraft(e.target.value)}
/> />
@@ -47,6 +62,7 @@ export function FilterBar({
</button> </button>
</div> </div>
{error && <p className="text-xs text-danger-text">{error}</p>} {error && <p className="text-xs text-danger-text">{error}</p>}
{!error && !draft && activeSummary && <p className="text-xs text-text-subtle">Active: {activeSummary}</p>}
</form> </form>
) )
} }
webapp/src/components/LogPreview.tsx
+10 -1
View File
@@ -209,7 +209,14 @@ function useScrollAnchor(containerRef: React.RefObject<HTMLDivElement | null>, d
}, deps) }, deps)
} }
export function LogPreview({ logs, params }: { logs: LogEntry[]; params: ParamDeclaration[] }) { export function LogPreview({
logs,
params,
}: {
logs: LogEntry[]
/** Ticket 045/053: already sorted by the caller's `fetchLogs` call per its own sort setting (owned and rendered up in `LogTimeline`'s header now) — this component just renders whatever order it's given. */
params: ParamDeclaration[]
}) {
const paramNames = params.map((d) => d.name) const paramNames = params.map((d) => d.name)
const [visibleParamColumns, setVisibleParamColumns] = useState<string[]>(loadVisibleParamColumns) const [visibleParamColumns, setVisibleParamColumns] = useState<string[]>(loadVisibleParamColumns)
const scrollContainerRef = useRef<HTMLDivElement>(null) const scrollContainerRef = useRef<HTMLDivElement>(null)
@@ -237,8 +244,10 @@ export function LogPreview({ logs, params }: { logs: LogEntry[]; params: ParamDe
Log preview Log preview
<HelpIcon sectionId="log-preview" label="Help: Log preview" /> <HelpIcon sectionId="log-preview" label="Help: Log preview" />
</div> </div>
<div className="flex items-center gap-3">
<ColumnPicker paramNames={paramNames} visible={visible} onToggle={toggleColumn} /> <ColumnPicker paramNames={paramNames} visible={visible} onToggle={toggleColumn} />
</div> </div>
</div>
<div ref={scrollContainerRef} className="flex-1 overflow-auto"> <div ref={scrollContainerRef} className="flex-1 overflow-auto">
<table className="w-full border-collapse text-left text-xs"> <table className="w-full border-collapse text-left text-xs">
<thead className="sticky top-0 bg-surface-0 text-text-subtle"> <thead className="sticky top-0 bg-surface-0 text-text-subtle">
webapp/src/components/LogTimeline.tsx
+113 -1
View File
@@ -41,18 +41,120 @@ function formatTimestamp(ms: number): string {
return new Date(ms).toLocaleTimeString() return new Date(ms).toLocaleTimeString()
} }
// Ticket 053: moved here from LogPreview.tsx, along with the `SortPicker` control below — the
// setting now renders in the Timeline's header, but it still drives `fetchLogs` up in App.tsx the
// same way it always did. Renamed from the old `ottyl.logPreview.sort` key since the control no
// longer lives in "log preview"; `loadLogSortSettings` still falls back to the old key so an
// existing choice survives this move instead of silently resetting on first load.
const LOG_SORT_STORAGE_KEY = 'ottyl.logTimeline.sort'
const LEGACY_LOG_SORT_STORAGE_KEY = 'ottyl.logPreview.sort'
export interface LogSortSettings {
/** An already-declared param name to order log lines by, or `null` for the default (`ottyl_ts`-tracking) order. */
sortBy: string | null
/** Only meaningful when `sortBy` is set. `true` (default) orders lines missing that param by `ottyl_seq`, appended after the sorted lines; `false` drops them from view. */
sortMissingFallback: boolean
}
const DEFAULT_LOG_SORT_SETTINGS: LogSortSettings = { sortBy: null, sortMissingFallback: true }
function parseLogSortSettings(raw: string | null): LogSortSettings | null {
if (!raw) return null
try {
const parsed = JSON.parse(raw)
if (!parsed || typeof parsed !== 'object') return null
const sortBy = typeof parsed.sortBy === 'string' ? parsed.sortBy : null
const sortMissingFallback = typeof parsed.sortMissingFallback === 'boolean' ? parsed.sortMissingFallback : true
return { sortBy, sortMissingFallback }
} catch {
return null
}
}
export function loadLogSortSettings(): LogSortSettings {
try {
return (
parseLogSortSettings(localStorage.getItem(LOG_SORT_STORAGE_KEY)) ??
parseLogSortSettings(localStorage.getItem(LEGACY_LOG_SORT_STORAGE_KEY)) ??
DEFAULT_LOG_SORT_SETTINGS
)
} catch {
return DEFAULT_LOG_SORT_SETTINGS
}
}
export function saveLogSortSettings(settings: LogSortSettings) {
try {
localStorage.setItem(LOG_SORT_STORAGE_KEY, JSON.stringify(settings))
} catch {
// best-effort — private browsing or full storage just means the selection won't persist across reloads
}
}
function SortPicker({
paramNames,
settings,
onChange,
}: {
paramNames: string[]
settings: LogSortSettings
onChange: (next: LogSortSettings) => void
}) {
return (
<div className="flex items-center gap-2 text-xs text-text-secondary">
<label className="flex items-center gap-1">
Sort by
<select
value={settings.sortBy ?? ''}
onChange={(e) => onChange({ ...settings, sortBy: e.target.value || null })}
className="rounded border border-border-soft bg-surface-0 px-1 py-0.5 text-text"
>
<option value="">ottyl_ts (default)</option>
{paramNames.map((name) => (
<option key={name} value={name}>
{name}
</option>
))}
</select>
</label>
{settings.sortBy && (
<label className="flex cursor-pointer items-center gap-1" title="How lines missing the sort param are handled">
<input
type="checkbox"
checked={settings.sortMissingFallback}
onChange={(e) => onChange({ ...settings, sortMissingFallback: e.target.checked })}
className="accent-text-muted"
/>
Fall back to ottyl_seq
</label>
)}
</div>
)
}
export function LogTimeline({ export function LogTimeline({
buckets, buckets,
range, range,
onSelectRange, onSelectRange,
activePreset, activePreset,
onSelectPreset, onSelectPreset,
paramNames,
sortSettings,
onSortSettingsChange,
sortIndexBuilding,
}: { }: {
buckets: TimelineBucket[] buckets: TimelineBucket[]
range: TimeRange | null range: TimeRange | null
onSelectRange: (range: TimeRange | null) => void onSelectRange: (range: TimeRange | null) => void
activePreset: TimeRangePresetId | null activePreset: TimeRangePresetId | null
onSelectPreset: (id: TimeRangePresetId) => void onSelectPreset: (id: TimeRangePresetId) => void
/** Ticket 045/053: current ordering choice, owned by the caller (it also feeds `fetchLogs`). Omitted by callers with no log list to sort (e.g. the standalone metrics/dashboards popout). */
paramNames?: string[]
sortSettings?: LogSortSettings
onSortSettingsChange?: (next: LogSortSettings) => void
/** Ticket 074: set by the caller when a `sort_by` fetch has been in flight long enough that it's
* likely the engine is running `build_sort_helper_index`'s one-time full scan, not just stalled. */
sortIndexBuilding?: boolean
}) { }) {
const containerRef = useRef<HTMLDivElement>(null) const containerRef = useRef<HTMLDivElement>(null)
const [dragStartX, setDragStartX] = useState<number | null>(null) const [dragStartX, setDragStartX] = useState<number | null>(null)
@@ -122,7 +224,16 @@ export function LogTimeline({
Timeline Timeline
<HelpIcon sectionId="timeline" label="Help: Timeline" /> <HelpIcon sectionId="timeline" label="Help: Timeline" />
</span> </span>
<div className="flex items-center gap-1.5 text-xs text-text-subtle"> <div className="flex items-center gap-3 text-xs text-text-subtle">
{sortSettings && onSortSettingsChange && (
<SortPicker paramNames={paramNames ?? []} settings={sortSettings} onChange={onSortSettingsChange} />
)}
{sortIndexBuilding && (
<span className="text-text-muted" title="First sort by this param scans every ingested line once; later polls reuse the built index.">
Building sort index…
</span>
)}
<div className="flex items-center gap-1.5">
<div className="flex items-center gap-1"> <div className="flex items-center gap-1">
{TIME_RANGE_PRESETS.map((preset) => ( {TIME_RANGE_PRESETS.map((preset) => (
<button <button
@@ -147,6 +258,7 @@ export function LogTimeline({
)} )}
</div> </div>
</div> </div>
</div>
<div className="p-3"> <div className="p-3">
<div <div
ref={containerRef} ref={containerRef}
webapp/src/components/Metrics.tsx
+100 -80
View File
@@ -3,13 +3,15 @@ import type { TimeRange, TimeRangePresetId } from './LogTimeline'
import { import {
ApiError, ApiError,
declareMetric, declareMetric,
deleteMetric,
fetchMetricBucketValues, fetchMetricBucketValues,
fetchMetricDeclarations, fetchMetricDeclarations,
fetchParamDeclarations,
metricPrimitiveToDisplayString, metricPrimitiveToDisplayString,
updateMetric, updateMetric,
type BucketValue, type BucketValue,
type MetricDeclaration, type MetricDeclaration,
type MetricParamMatcher, type ParamDeclaration,
} from '../lib/api' } from '../lib/api'
import { import {
CollapseToggle, CollapseToggle,
@@ -22,8 +24,6 @@ import {
import { HelpIcon } from './HelpIcon' import { HelpIcon } from './HelpIcon'
import { RegexHelpIcon } from './RegexHelpIcon' import { RegexHelpIcon } from './RegexHelpIcon'
const EMPTY_MATCHER: MetricParamMatcher = { param_name: '', capture_group: '', kind: 'string' }
// One row in the labels -> occurrence count table; `values` keeps the raw points for the sparkline rendering. // One row in the labels -> occurrence count table; `values` keeps the raw points for the sparkline rendering.
interface LabelGroup { interface LabelGroup {
key: string key: string
@@ -80,12 +80,16 @@ function sparkline(values: BucketValue[], range: TimeRange | null): string {
export function Metrics({ range, activePreset }: { range: TimeRange | null; activePreset?: TimeRangePresetId | null }) { export function Metrics({ range, activePreset }: { range: TimeRange | null; activePreset?: TimeRangePresetId | null }) {
const [metrics, setMetrics] = useState<MetricDeclaration[]>([]) const [metrics, setMetrics] = useState<MetricDeclaration[]>([])
const [params, setParams] = useState<ParamDeclaration[]>([])
const [selected, setSelected] = useState<string | null>(null) const [selected, setSelected] = useState<string | null>(null)
const [bucketValues, setBucketValues] = useState<BucketValue[]>([]) const [bucketValues, setBucketValues] = useState<BucketValue[]>([])
const [name, setName] = useState('') const [name, setName] = useState('')
const [regex, setRegex] = useState('') const [regex, setRegex] = useState('')
const [matchers, setMatchers] = useState<MetricParamMatcher[]>([{ ...EMPTY_MATCHER }]) // A metric picks its label/value params from the params already declared in the Params panel
// (ticket 037) rather than re-entering a regex/capture-group/kind per param.
const [labelParams, setLabelParams] = useState<string[]>([])
const [valueParam, setValueParam] = useState<string>('')
const [submitError, setSubmitError] = useState<string | null>(null) const [submitError, setSubmitError] = useState<string | null>(null)
const [submitting, setSubmitting] = useState(false) const [submitting, setSubmitting] = useState(false)
const [collapsed, setCollapsed] = useSectionCollapsed('metrics') const [collapsed, setCollapsed] = useSectionCollapsed('metrics')
@@ -101,8 +105,17 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
} }
} }
async function loadParams() {
try {
setParams(await fetchParamDeclarations())
} catch {
// engine not reachable yet; leave the previous list in place
}
}
useEffect(() => { useEffect(() => {
loadMetrics() loadMetrics()
loadParams()
}, []) }, [])
// Refetch-on-change rather than continuous polling: bucket values only move when the selected metric or range changes, so no interval is needed here like `App.tsx`'s log/timeline polling. // Refetch-on-change rather than continuous polling: bucket values only move when the selected metric or range changes, so no interval is needed here like `App.tsx`'s log/timeline polling.
@@ -124,23 +137,18 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
} }
}, [selected, range]) }, [selected, range])
function updateMatcher(index: number, patch: Partial<MetricParamMatcher>) { function toggleLabelParam(paramName: string) {
setMatchers((prev) => prev.map((m, i) => (i === index ? { ...m, ...patch } : m))) setLabelParams((prev) =>
} prev.includes(paramName) ? prev.filter((name) => name !== paramName) : [...prev, paramName],
)
function addMatcher() {
setMatchers((prev) => [...prev, { ...EMPTY_MATCHER }])
}
function removeMatcher(index: number) {
setMatchers((prev) => prev.filter((_, i) => i !== index))
} }
function resetForm() { function resetForm() {
setEditingName(null) setEditingName(null)
setName('') setName('')
setRegex('') setRegex('')
setMatchers([{ ...EMPTY_MATCHER }]) setLabelParams([])
setValueParam('')
setSubmitError(null) setSubmitError(null)
} }
@@ -149,7 +157,8 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
setEditingName(metric.name) setEditingName(metric.name)
setName(metric.name) setName(metric.name)
setRegex(metric.message_match_regex) setRegex(metric.message_match_regex)
setMatchers(metric.params.length > 0 ? metric.params.map((p) => ({ ...p })) : [{ ...EMPTY_MATCHER }]) setLabelParams(metric.label_params)
setValueParam(metric.value_param ?? '')
setSubmitError(null) setSubmitError(null)
} }
@@ -158,6 +167,19 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
editMetric(metric) editMetric(metric)
} }
// No confirm-before-delete dialog — matches the rest of this settings UI (e.g. CSS entry / profile
// deletion in NavMenu.tsx), which also deletes directly on click rather than prompting first.
async function handleDelete(metricName: string) {
try {
await deleteMetric(metricName)
if (selected === metricName) setSelected(null)
if (editingName === metricName) resetForm()
await loadMetrics()
} catch {
// best-effort; the declared-metrics list simply keeps showing it if the delete failed
}
}
async function handleSubmit(e: React.FormEvent) { async function handleSubmit(e: React.FormEvent) {
e.preventDefault() e.preventDefault()
setSubmitError(null) setSubmitError(null)
@@ -166,7 +188,8 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
const request: MetricDeclaration = { const request: MetricDeclaration = {
name, name,
message_match_regex: regex, message_match_regex: regex,
params: matchers.filter((m) => m.param_name && m.capture_group), label_params: labelParams,
value_param: valueParam || null,
} }
const saved = editingName ? await updateMetric(editingName, request) : await declareMetric(request) const saved = editingName ? await updateMetric(editingName, request) : await declareMetric(request)
resetForm() resetForm()
@@ -227,61 +250,49 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
/> />
</div> </div>
<div> <div>
<div className="mb-1 flex items-center justify-between text-xs text-text-subtle"> <label className="mb-1 block text-xs text-text-subtle">
<span>Params</span> Label params
<button {params.length === 0 && (
type="button" <span className="ml-1 text-text-faint">(declare a param in the Params panel below first)</span>
onClick={addMatcher} )}
className="rounded border border-border-soft px-2 py-0.5 text-text-secondary hover:bg-surface-2" </label>
> <div className="max-h-32 space-y-0.5 overflow-auto rounded border border-border-soft bg-surface-1 p-1.5">
+ Add {params.length === 0 ? (
</button> <span className="text-xs text-text-faint">No params declared yet.</span>
</div> ) : (
<div className="space-y-1"> params.map((p) => (
{matchers.map((matcher, i) => ( <label key={p.name} className="flex items-center gap-1.5 text-xs text-text-secondary">
<div key={i} className="flex items-center gap-1">
<input <input
className="w-24 min-w-0 flex-1 rounded border border-border-soft bg-surface-1 px-1.5 py-1 text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none" type="checkbox"
value={matcher.param_name} checked={labelParams.includes(p.name)}
onChange={(e) => updateMatcher(i, { param_name: e.target.value })} onChange={() => toggleLabelParam(p.name)}
placeholder="param_name"
/> />
<input <span className="font-mono">{p.name}</span>
className="w-24 min-w-0 flex-1 rounded border border-border-soft bg-surface-1 px-1.5 py-1 text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none" <span className="text-text-faint">({p.elapsed_since_last ? 'elapsed since previous' : p.kind})</span>
value={matcher.capture_group} </label>
onChange={(e) => updateMatcher(i, { capture_group: e.target.value })} ))
placeholder="capture_group" )}
/>
<select
className="rounded border border-border-soft bg-surface-1 px-1 py-1 text-xs text-text focus:border-border-strong focus:outline-none"
value={matcher.kind}
onChange={(e) => updateMatcher(i, { kind: e.target.value as MetricParamMatcher['kind'] })}
>
<option value="string">string</option>
<option value="number">number</option>
<option value="timestamp">timestamp</option>
<option value="duration">duration</option>
</select>
<select
className="rounded border border-border-soft bg-surface-1 px-1 py-1 text-xs text-text focus:border-border-strong focus:outline-none"
value={matcher.role ?? 'label'}
onChange={(e) => updateMatcher(i, { role: e.target.value as MetricParamMatcher['role'] })}
title="Label params identify a bucket; at most one Value param becomes the counter's per-event captured value (used by histogram()/gauge()). A Value param of kind 'duration' is derived automatically — elapsed time since the previous match with the same labels — rather than parsed from the capture group."
>
<option value="label">label</option>
<option value="value">value</option>
</select>
<button
type="button"
onClick={() => removeMatcher(i)}
disabled={matchers.length === 1}
className="rounded border border-border-soft px-1.5 py-1 text-xs text-text-muted hover:bg-surface-2 disabled:opacity-30"
>
×
</button>
</div> </div>
</div>
<div>
<label
className="mb-1 block text-xs text-text-subtle"
title="At most one param becomes the counter's per-event captured value (used by histogram()/gauge()) instead of a label. An 'elapsed since previous event' param (declared in the Params panel) is computed automatically — elapsed time since the previous match with the same labels — rather than parsed from its capture group."
>
Value param
</label>
<select
className="w-full rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text focus:border-border-strong focus:outline-none"
value={valueParam}
onChange={(e) => setValueParam(e.target.value)}
>
<option value="">(none — occurrence count)</option>
{params.map((p) => (
<option key={p.name} value={p.name}>
{p.name} ({p.elapsed_since_last ? 'elapsed since previous' : p.kind})
</option>
))} ))}
</div> </select>
</div> </div>
{editingName && ( {editingName && (
<div className="rounded border border-accent bg-accent/5 px-2 py-1 text-xs text-accent-text"> <div className="rounded border border-accent bg-accent/5 px-2 py-1 text-xs text-accent-text">
@@ -315,18 +326,27 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
<div className="mb-3 flex flex-wrap gap-1"> <div className="mb-3 flex flex-wrap gap-1">
{metrics.length === 0 && <span className="text-xs text-text-faint">No metrics declared yet.</span>} {metrics.length === 0 && <span className="text-xs text-text-faint">No metrics declared yet.</span>}
{metrics.map((m) => ( {metrics.map((m) => (
<button <div
key={m.name} key={m.name}
onClick={() => selectMetric(m)} className={`flex items-center gap-0.5 rounded border pl-2 pr-1 py-1 text-xs ${
className={`rounded border px-2 py-1 text-xs ${
selected === m.name selected === m.name
? 'border-accent bg-accent/10 text-accent-text' ? 'border-accent bg-accent/10 text-accent-text'
: 'border-border-soft text-text-secondary hover:bg-surface-2' : 'border-border-soft text-text-secondary hover:bg-surface-2'
}`} }`}
title={m.message_match_regex}
> >
{m.name} ({m.params.map((p) => p.param_name).join(', ') || 'no params'}) <button type="button" onClick={() => selectMetric(m)} title={m.message_match_regex}>
{m.name} ({m.label_params.join(', ') || 'no params'})
</button> </button>
<button
type="button"
onClick={() => handleDelete(m.name)}
className="rounded px-1 text-text-faint hover:bg-danger-bg/50 hover:text-danger-text"
aria-label={`Delete metric ${m.name}`}
title={`Delete metric "${m.name}"`}
>
×
</button>
</div>
))} ))}
</div> </div>
@@ -342,17 +362,17 @@ export function Metrics({ range, activePreset }: { range: TimeRange | null; acti
<span className="font-mono">{selectedMetric.message_match_regex}</span> <span className="font-mono">{selectedMetric.message_match_regex}</span>
</div> </div>
<div> <div>
<span className="text-text-subtle">Params: </span> <span className="text-text-subtle">Label params: </span>
{selectedMetric.params.length === 0 ? ( {selectedMetric.label_params.length === 0 ? (
<span className="text-text-faint">none</span> <span className="text-text-faint">none</span>
) : ( ) : (
<span className="font-mono"> <span className="font-mono">{selectedMetric.label_params.join(', ')}</span>
{selectedMetric.params
.map((p) => `${p.param_name}←${p.capture_group} (${p.kind}, ${p.role ?? 'label'})`)
.join(', ')}
</span>
)} )}
</div> </div>
<div>
<span className="text-text-subtle">Value param: </span>
<span className="font-mono">{selectedMetric.value_param ?? <span className="text-text-faint">none</span>}</span>
</div>
</div> </div>
)} )}
{groups.length === 0 ? ( {groups.length === 0 ? (
webapp/src/components/NavMenu.tsx
+148 -115
View File
@@ -6,6 +6,7 @@ import { useDocsNav, type DocsTarget } from '../lib/docsNav'
import { DocsPanel } from './Docs' import { DocsPanel } from './Docs'
import { HelpIcon } from './HelpIcon' import { HelpIcon } from './HelpIcon'
import { BUILTIN_THEMES, downloadThemeFile, generateThemeCss, parseThemeCss } from '../lib/themes' import { BUILTIN_THEMES, downloadThemeFile, generateThemeCss, parseThemeCss } from '../lib/themes'
import ottylIcon from '../assets/ottyl-icon.svg'
type PanelName = 'settings' | 'docs' | null type PanelName = 'settings' | 'docs' | null
@@ -27,8 +28,29 @@ const COLOR_SCHEMES: ColorScheme[] = [
'yellow', 'yellow',
'pink', 'pink',
'orange', 'orange',
'light_green',
'light_red',
'light_blue',
'light_yellow',
'light_pink',
'light_orange',
] ]
/** Display label for a `ColorScheme`: underscores become spaces so `capitalize` (which titles each word) renders e.g. "light_green" as "Light Green". */
function schemeLabel(scheme: ColorScheme): string {
return scheme.replace(/_/g, ' ')
}
/** Ticket 073: plain `Xms` under a second (this stat is normally near-zero, so that's the common case), a humantime-style `Xs`/`Xm` above it — mirrors the backend's own humantime-duration inputs (see `Docs.tsx`'s window-parsing note) without pulling in a full duration-formatting library for one stat. */
function formatIngestLag(ms: number): string {
if (ms < 1000) return `${ms}ms`
const totalSeconds = ms / 1000
if (totalSeconds < 60) return `${totalSeconds % 1 === 0 ? totalSeconds : totalSeconds.toFixed(1)}s`
const minutes = Math.floor(totalSeconds / 60)
const seconds = Math.round(totalSeconds % 60)
return seconds === 0 ? `${minutes}m` : `${minutes}m${seconds}s`
}
export function NavMenu({ stats }: { stats: Stats | null }) { export function NavMenu({ stats }: { stats: Stats | null }) {
const [openPanel, setOpenPanel] = useState<PanelName>(null) const [openPanel, setOpenPanel] = useState<PanelName>(null)
const panel = useSettingsPanel() const panel = useSettingsPanel()
@@ -56,7 +78,10 @@ export function NavMenu({ stats }: { stats: Stats | null }) {
<div className="flex h-full"> <div className="flex h-full">
<div className="flex h-full w-48 shrink-0 flex-col justify-between border-r border-border p-3"> <div className="flex h-full w-48 shrink-0 flex-col justify-between border-r border-border p-3">
<div> <div>
<div className="mb-4 text-lg font-semibold text-text">ottyl</div> <div className="mb-4 flex items-center gap-2 text-lg font-semibold text-text">
<img src={ottylIcon} alt="" className="h-5 w-5" />
ottyl
</div>
<PresetsDropdown panel={panel} /> <PresetsDropdown panel={panel} />
<nav className="mt-2 flex flex-col gap-1 text-sm text-text-muted"> <nav className="mt-2 flex flex-col gap-1 text-sm text-text-muted">
<button <button
@@ -87,6 +112,9 @@ export function NavMenu({ stats }: { stats: Stats | null }) {
seq {stats.first_sequence ?? 0}–{stats.last_sequence ?? 0} seq {stats.first_sequence ?? 0}–{stats.last_sequence ?? 0}
</div> </div>
<div>{stats.approx_bytes_total.toLocaleString()} bytes</div> <div>{stats.approx_bytes_total.toLocaleString()} bytes</div>
<div title="Highest ingestion-to-engine-processing lag seen in the last 5s">
lag {formatIngestLag(stats.ingest_lag_high_watermark_ms)}
</div>
</> </>
) : ( ) : (
<div>connecting…</div> <div>connecting…</div>
@@ -252,8 +280,10 @@ function SettingsPanel({ panel }: { panel: ReturnType<typeof useSettingsPanel> }
setCssUploadError(null) setCssUploadError(null)
} }
// Every imported/linked stylesheet is always a selectable theme (ticket 020), so it's always // Every imported/linked stylesheet is always a selectable style, listed alongside the built-in
// validated against the canonical `--c-*` variable set before it's sent. // colour schemes (ticket 020/047), so it's always validated against the canonical `--c-*`
// variable set before it's sent — that's a floor, not a ceiling: the stylesheet is free to also
// carry rules for any other selector, up to and including a full layout redesign.
async function handleCssImport(e: React.FormEvent) { async function handleCssImport(e: React.FormEvent) {
e.preventDefault() e.preventDefault()
setCssUploadError(null) setCssUploadError(null)
@@ -294,8 +324,14 @@ function SettingsPanel({ panel }: { panel: ReturnType<typeof useSettingsPanel> }
return ( return (
<div className="flex flex-col gap-5 text-sm text-text-secondary"> <div className="flex flex-col gap-5 text-sm text-text-secondary">
<section> <section>
<h3 className="mb-2 text-xs font-semibold uppercase tracking-wide text-text-subtle">Colour scheme</h3> <div className="mb-2 flex items-center gap-1.5">
<div className="flex flex-wrap gap-1"> <h3 className="text-xs font-semibold uppercase tracking-wide text-text-subtle">Style</h3>
<span className="text-[11px] text-text-faint">
one list of interchangeable styles — a built-in palette or your own imported/linked stylesheet. Any of
them can be a simple colour swap or a full layout redesign; only one is active at a time.
</span>
</div>
<div className="mb-3 flex flex-wrap gap-1">
{COLOR_SCHEMES.map((scheme) => ( {COLOR_SCHEMES.map((scheme) => (
<div key={scheme} className="flex items-center"> <div key={scheme} className="flex items-center">
<button <button
@@ -307,14 +343,14 @@ function SettingsPanel({ panel }: { panel: ReturnType<typeof useSettingsPanel> }
: 'border-border-soft text-text-muted hover:bg-surface-2' : 'border-border-soft text-text-muted hover:bg-surface-2'
}`} }`}
> >
{scheme} {schemeLabel(scheme)}
</button> </button>
{scheme !== 'system' && ( {scheme !== 'system' && (
<button <button
type="button" type="button"
onClick={() => downloadBuiltinTheme(scheme)} onClick={() => downloadBuiltinTheme(scheme)}
className="rounded-r border border-l-0 border-border-soft px-1.5 py-1 text-xs text-text-faint hover:bg-surface-2 hover:text-text" className="rounded-r border border-l-0 border-border-soft px-1.5 py-1 text-xs text-text-faint hover:bg-surface-2 hover:text-text"
aria-label={`Download ${scheme} theme`} aria-label={`Download ${schemeLabel(scheme)} theme`}
title="Download as .css" title="Download as .css"
> >
↓ ↓
@@ -323,6 +359,111 @@ function SettingsPanel({ panel }: { panel: ReturnType<typeof useSettingsPanel> }
</div> </div>
))} ))}
</div> </div>
{panel.cssEntries.length === 0 ? (
<div className="mb-2 text-xs text-text-faint">No styles imported or linked yet.</div>
) : (
<ul className="mb-2 flex flex-col gap-1">
{panel.cssEntries.map((entry) => (
<li
key={entry.id}
className="flex items-center justify-between rounded border border-border px-2 py-1 text-xs"
>
<div className="min-w-0 flex-1">
<button
type="button"
onClick={() => panel.setCustomTheme(entry.id)}
className={`block w-full truncate text-left capitalize ${
panel.activeCustomTheme?.id === entry.id ? 'text-text' : 'text-text-muted hover:text-text'
}`}
>
{entry.filename ?? entry.path_or_url}
</button>
<div className="text-[10px] uppercase text-text-faint">{entry.kind}</div>
</div>
<div className="ml-2 flex shrink-0 items-center gap-1">
<button
type="button"
onClick={() => downloadCssEntry(entry)}
className="rounded px-1.5 text-text-subtle hover:bg-surface-2 hover:text-text"
aria-label={`Download ${entry.filename ?? entry.path_or_url}`}
title="Download as .css"
>
↓
</button>
{entry.kind === 'imported' ? (
<button
type="button"
onClick={() => panel.removeCssEntry(entry.id)}
className="rounded px-1.5 text-text-subtle hover:bg-surface-2 hover:text-danger-text"
aria-label={`Delete ${entry.filename}`}
>
delete
</button>
) : (
<span className="text-text-faintest" title="Linked CSS isn't deletable here">
—
</span>
)}
</div>
</li>
))}
</ul>
)}
<p className="mb-2 text-[11px] text-text-faint">
Add your own style below — a stylesheet, uploaded or linked by URL/path. It must at least define every{' '}
<code>--c-*</code> variable the built-ins do, but nothing stops it from also targeting any other selector
in the app (layout, spacing, fonts, anything) — it isn't limited to palette overrides.
</p>
<form onSubmit={handleCssImport} className="flex flex-col gap-2">
<input
type="file"
accept=".css,text/css"
onChange={handleCssFileChange}
className="text-xs text-text-muted file:mr-2 file:rounded file:border file:border-border-soft file:bg-surface-2 file:px-2 file:py-1 file:text-text"
/>
<input
type="text"
placeholder="enter name"
value={cssFilename}
onChange={(e) => setCssFilename(e.target.value)}
className="rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none"
/>
<textarea
placeholder="Paste CSS content here"
value={cssContent}
onChange={(e) => setCssContent(e.target.value)}
rows={4}
className="resize-y rounded border border-border-soft bg-surface-1 px-2 py-1 font-mono text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none"
/>
{cssUploadError && <div className="text-xs text-danger-text-strong">{cssUploadError}</div>}
<button
type="submit"
className="rounded border border-border-soft px-2 py-1 text-xs text-text hover:bg-surface-2"
>
Import
</button>
</form>
<form onSubmit={handleCssLink} className="mt-2 flex flex-col gap-2">
<input
type="text"
placeholder="https://example.com/style.css"
value={cssLinkUrl}
onChange={(e) => setCssLinkUrl(e.target.value)}
className="min-w-0 flex-1 rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none"
/>
<div className="flex items-center gap-2">
<button
type="submit"
className="shrink-0 rounded border border-border-soft px-2 py-1 text-xs text-text hover:bg-surface-2"
>
Link
</button>
</div>
</form>
</section> </section>
<section> <section>
@@ -378,114 +519,6 @@ function SettingsPanel({ panel }: { panel: ReturnType<typeof useSettingsPanel> }
</button> </button>
</div> </div>
</section> </section>
<section>
<div className="mb-2 flex items-center gap-1.5">
<h3 className="text-xs font-semibold uppercase tracking-wide text-text-subtle">CSS</h3>
<span className="text-[11px] text-text-faint">
imported/linked stylesheets — each one becomes a selectable alternative to the built-in colour schemes
above
</span>
</div>
{panel.cssEntries.length === 0 ? (
<div className="mb-2 text-xs text-text-faint">No CSS configured.</div>
) : (
<ul className="mb-2 flex flex-col gap-1">
{panel.cssEntries.map((entry) => (
<li
key={entry.id}
className="flex items-center justify-between rounded border border-border px-2 py-1 text-xs"
>
<div className="min-w-0 flex-1">
<button
type="button"
onClick={() => panel.setCustomTheme(entry.id)}
className={`block w-full truncate text-left capitalize ${
panel.activeCustomTheme?.id === entry.id ? 'text-text' : 'text-text-muted hover:text-text'
}`}
>
{entry.filename ?? entry.path_or_url}
</button>
<div className="text-[10px] uppercase text-text-faint">{entry.kind}</div>
</div>
<div className="ml-2 flex shrink-0 items-center gap-1">
<button
type="button"
onClick={() => downloadCssEntry(entry)}
className="rounded px-1.5 text-text-subtle hover:bg-surface-2 hover:text-text"
aria-label={`Download ${entry.filename ?? entry.path_or_url}`}
title="Download as .css"
>
↓
</button>
{entry.kind === 'imported' ? (
<button
type="button"
onClick={() => panel.removeCssEntry(entry.id)}
className="rounded px-1.5 text-text-subtle hover:bg-surface-2 hover:text-danger-text"
aria-label={`Delete ${entry.filename}`}
>
delete
</button>
) : (
<span className="text-text-faintest" title="Linked CSS isn't deletable here">
—
</span>
)}
</div>
</li>
))}
</ul>
)}
<form onSubmit={handleCssImport} className="flex flex-col gap-2">
<input
type="file"
accept=".css,text/css"
onChange={handleCssFileChange}
className="text-xs text-text-muted file:mr-2 file:rounded file:border file:border-border-soft file:bg-surface-2 file:px-2 file:py-1 file:text-text"
/>
<input
type="text"
placeholder="enter name"
value={cssFilename}
onChange={(e) => setCssFilename(e.target.value)}
className="rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none"
/>
<textarea
placeholder="Paste CSS content here"
value={cssContent}
onChange={(e) => setCssContent(e.target.value)}
rows={4}
className="resize-y rounded border border-border-soft bg-surface-1 px-2 py-1 font-mono text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none"
/>
{cssUploadError && <div className="text-xs text-danger-text-strong">{cssUploadError}</div>}
<button
type="submit"
className="rounded border border-border-soft px-2 py-1 text-xs text-text hover:bg-surface-2"
>
Import
</button>
</form>
<form onSubmit={handleCssLink} className="mt-2 flex flex-col gap-2">
<input
type="text"
placeholder="https://example.com/style.css"
value={cssLinkUrl}
onChange={(e) => setCssLinkUrl(e.target.value)}
className="min-w-0 flex-1 rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none"
/>
<div className="flex items-center gap-2">
<button
type="submit"
className="shrink-0 rounded border border-border-soft px-2 py-1 text-xs text-text hover:bg-surface-2"
>
Link
</button>
</div>
</form>
</section>
</div> </div>
) )
} }
webapp/src/components/PanelChart.tsx
+53 -5
View File
@@ -14,6 +14,7 @@ import {
YAxis, YAxis,
} from 'recharts' } from 'recharts'
import type { ChartType, PromqlSeries } from '../lib/api' import type { ChartType, PromqlSeries } from '../lib/api'
import type { TimeRange } from './LogTimeline'
/// Renders a label combination as a series key/legend string, e.g. `{"path":"/a"}` -> `path=/a`; no labels renders as `value`. /// Renders a label combination as a series key/legend string, e.g. `{"path":"/a"}` -> `path=/a`; no labels renders as `value`.
export function seriesLabel(labels: Record<string, string>): string { export function seriesLabel(labels: Record<string, string>): string {
@@ -22,8 +23,38 @@ export function seriesLabel(labels: Record<string, string>): string {
return entries.map(([k, v]) => `${k}=${v}`).join(', ') return entries.map(([k, v]) => `${k}=${v}`).join(', ')
} }
// Fixed, small categorical palette to avoid a colour-scale dependency; cycles if there are more series than colours. // Categorical palette drawn from the active theme's accent set (ticket 048: every built-in theme
const COLORS = ['#38bdf8', '#f472b6', '#a78bfa', '#4ade80', '#fbbf24', '#fb7185', '#2dd4bf', '#f97316'] // exposes --c-accent/--c-accent-2/--c-accent-3, not just one accent hue) instead of a hardcoded,
// theme-independent hex list. `color-mix` tints/shades those three base accents into a fuller
// 8-colour cycle (matching the previous fixed palette's size, so MAX_KEPT_SERIES/squashing below is
// unaffected) without any JS colour math or a getComputedStyle read — the browser resolves the CSS
// variables per-theme at paint time, same as `stroke="var(--c-border)"` elsewhere in this file.
const COLORS = [
'var(--c-accent)',
'var(--c-accent-2)',
'var(--c-accent-3)',
'color-mix(in srgb, var(--c-accent) 60%, white)',
'color-mix(in srgb, var(--c-accent-2) 60%, white)',
'color-mix(in srgb, var(--c-accent-3) 60%, white)',
'color-mix(in srgb, var(--c-accent) 55%, black)',
'color-mix(in srgb, var(--c-accent-2) 55%, black)',
]
// One less than the palette size: above this many series, keep the top MAX_KEPT_SERIES (by value) and squash the
// rest into a single "Other" bucket, so the squashed range never exceeds COLORS.length and never repeats a colour.
const MAX_KEPT_SERIES = COLORS.length - 1
// Sorts rows by value descending and, if there are more than MAX_KEPT_SERIES, keeps the top MAX_KEPT_SERIES and
// collapses the remainder into a single "Other" row summing their values. Below the threshold, rows are returned
// unchanged so panels with few series keep their existing bar/slice order and colour assignment.
function squashRows(rows: { name: string; value: number }[]): { name: string; value: number }[] {
if (rows.length <= MAX_KEPT_SERIES) return rows
const sorted = [...rows].sort((a, b) => b.value - a.value)
const kept = sorted.slice(0, MAX_KEPT_SERIES)
const rest = sorted.slice(MAX_KEPT_SERIES)
const otherValue = rest.reduce((sum, r) => sum + r.value, 0)
return [...kept, { name: `Other (${rest.length})`, value: otherValue }]
}
// Merges series into one row-per-timestamp dataset (recharts wants a single array keyed by series name); a timestamp missing from a series is left undefined there, which recharts skips when drawing that line. // Merges series into one row-per-timestamp dataset (recharts wants a single array keyed by series name); a timestamp missing from a series is left undefined there, which recharts skips when drawing that line.
function toLineData(series: PromqlSeries[]): { rows: Record<string, number | string>[]; keys: string[] } { function toLineData(series: PromqlSeries[]): { rows: Record<string, number | string>[]; keys: string[] } {
@@ -61,19 +92,36 @@ function sumValue(series: PromqlSeries): number {
return series.points.reduce((total, [, value]) => total + value, 0) return series.points.reduce((total, [, value]) => total + value, 0)
} }
export function PanelChart({ series, chartType }: { series: PromqlSeries[]; chartType: ChartType }) { export function PanelChart({
series,
chartType,
range,
}: {
series: PromqlSeries[]
chartType: ChartType
range?: TimeRange | null
}) {
if (series.length === 0) { if (series.length === 0) {
return <div className="flex h-48 items-center justify-center text-xs text-text-faint">No data in range.</div> return <div className="flex h-48 items-center justify-center text-xs text-text-faint">No data in range.</div>
} }
if (chartType === 'line') { if (chartType === 'line') {
const { rows, keys } = toLineData(series) const { rows, keys } = toLineData(series)
// Ticket 057: without an explicit numeric domain, recharts auto-scales the x-axis to the
// min/max timestamp actually present in `rows` — so a metric whose occurrences only cover a
// sliver of the queried range (a burst followed by silence) silently renders as if that
// sliver *were* the full range, with no visual sign the axis got narrowed. Pinning the domain
// to the panel's actual queried range keeps the axis honest; `type="number"` is required for
// a `domain` tuple to be interpreted as a value range rather than a category filter.
return ( return (
<ResponsiveContainer width="100%" height={220}> <ResponsiveContainer width="100%" height={220}>
<LineChart data={rows} margin={{ top: 8, right: 16, left: 0, bottom: 0 }}> <LineChart data={rows} margin={{ top: 8, right: 16, left: 0, bottom: 0 }}>
<CartesianGrid strokeDasharray="3 3" stroke="var(--c-border)" /> <CartesianGrid strokeDasharray="3 3" stroke="var(--c-border)" />
<XAxis <XAxis
dataKey="timestamp_ms" dataKey="timestamp_ms"
type="number"
domain={range ? [range.startMs, range.endMs] : ['dataMin', 'dataMax']}
allowDataOverflow
tickFormatter={formatTimestamp} tickFormatter={formatTimestamp}
stroke="var(--c-text-subtle)" stroke="var(--c-text-subtle)"
tick={{ fontSize: 11, fill: 'var(--c-text-muted)' }} tick={{ fontSize: 11, fill: 'var(--c-text-muted)' }}
@@ -101,7 +149,7 @@ export function PanelChart({ series, chartType }: { series: PromqlSeries[]; char
} }
if (chartType === 'bar') { if (chartType === 'bar') {
const rows = series.map((s) => ({ name: seriesLabel(s.labels), value: latestValue(s) })) const rows = squashRows(series.map((s) => ({ name: seriesLabel(s.labels), value: latestValue(s) })))
return ( return (
<ResponsiveContainer width="100%" height={220}> <ResponsiveContainer width="100%" height={220}>
<BarChart data={rows} margin={{ top: 8, right: 16, left: 0, bottom: 0 }}> <BarChart data={rows} margin={{ top: 8, right: 16, left: 0, bottom: 0 }}>
@@ -119,7 +167,7 @@ export function PanelChart({ series, chartType }: { series: PromqlSeries[]; char
) )
} }
const rows = series.map((s) => ({ name: seriesLabel(s.labels), value: sumValue(s) })) const rows = squashRows(series.map((s) => ({ name: seriesLabel(s.labels), value: sumValue(s) })))
const total = rows.reduce((sum, r) => sum + r.value, 0) const total = rows.reduce((sum, r) => sum + r.value, 0)
if (total === 0) { if (total === 0) {
return <div className="flex h-48 items-center justify-center text-xs text-text-faint">No non-zero values in range.</div> return <div className="flex h-48 items-center justify-center text-xs text-text-faint">No non-zero values in range.</div>
webapp/src/components/Params.tsx
+185 -34
View File
@@ -1,5 +1,5 @@
import { useMemo, useState } from 'react' import { useMemo, useState } from 'react'
import { ApiError, declareParam, type ParamDeclaration } from '../lib/api' import { ApiError, clearParams, declareParam, type ParamDeclaration } from '../lib/api'
import { import {
CollapseToggle, CollapseToggle,
handleSectionHeaderKeyDown, handleSectionHeaderKeyDown,
@@ -13,7 +13,11 @@ import { RegexHelpIcon } from './RegexHelpIcon'
const EMPTY_KIND: ParamDeclaration['kind'] = 'string' const EMPTY_KIND: ParamDeclaration['kind'] = 'string'
type ExtractionSource = 'regex' | 'json_field' // 'elapsed_since_last' is always offered, regardless of what the message match regex looks like —
// unlike 'json_field' (which only makes sense once the sample line looks like a JSON object), this
// param's value is never extracted from text at all: it's elapsed time since the previous event
// sharing the same label set (ticket 038).
type ExtractionSource = 'regex' | 'json_field' | 'elapsed_since_last'
/// A JSON *object* (not an array/scalar) parses cleanly and pretty-prints with braces — the same /// A JSON *object* (not an array/scalar) parses cleanly and pretty-prints with braces — the same
/// shape a JSON field path can be read from, so this is the "line looks like JSON" gate that /// shape a JSON field path can be read from, so this is the "line looks like JSON" gate that
@@ -39,40 +43,116 @@ export function Params({ params, onDeclared }: { params: ParamDeclaration[]; onD
const [submitError, setSubmitError] = useState<string | null>(null) const [submitError, setSubmitError] = useState<string | null>(null)
const [submitting, setSubmitting] = useState(false) const [submitting, setSubmitting] = useState(false)
const [collapsed, setCollapsed] = useSectionCollapsed('params') const [collapsed, setCollapsed] = useSectionCollapsed('params')
// Name of the declaration currently loaded into the form for editing, or `null` when the form is
// declaring a brand-new param. The name field is locked while editing — `declareParam` overwrites
// in place by name, so changing it here would silently create a second param and orphan the
// original (and anything referencing it by name) rather than renaming it.
const [editingName, setEditingName] = useState<string | null>(null)
const [clearing, setClearing] = useState(false)
const [clearResult, setClearResult] = useState<string | null>(null)
const [clearError, setClearError] = useState<string | null>(null)
// `reserved` params (`ottyl_seq`/`ottyl_ts`/`ottyl_fmt`) are synthesized server-side purely for
// discoverability (see `GET /api/params`) — they aren't real declarations, so they're kept out of
// the editable/clickable list and shown separately, non-interactively.
const declaredParams = useMemo(() => params.filter((p) => !p.reserved), [params])
const reservedParams = useMemo(() => params.filter((p) => p.reserved), [params])
const isBoolean = kind === 'boolean' const isBoolean = kind === 'boolean'
const isElapsedSinceLast = source === 'elapsed_since_last'
// The message match regex doubles as a sample line here so a pasted JSON log line unlocks the // The message match regex doubles as a sample line here so a pasted JSON log line unlocks the
// "JSON field" extraction option — regex-based params are unaffected either way. // "JSON field" extraction option — regex-based params are unaffected either way. Not consulted at
// all for "elapsed since previous event", which is always offered.
const detectedJson = useMemo(() => looksLikeJsonObject(regex), [regex]) const detectedJson = useMemo(() => looksLikeJsonObject(regex), [regex])
const effectiveSource: ExtractionSource = detectedJson && !isBoolean ? source : 'regex' const effectiveSource: ExtractionSource = isElapsedSinceLast
? 'elapsed_since_last'
: detectedJson && !isBoolean
? source
: 'regex'
async function handleSubmit(e: React.FormEvent) { function handleSourceChange(next: ExtractionSource) {
e.preventDefault() setSource(next)
setSubmitError(null) // "Elapsed since previous event" always produces a Duration value — never text-extracted — so
setSubmitting(true) // there's no separate kind to pick; force it rather than leaving a stale/mismatched kind selected.
try { if (next === 'elapsed_since_last') setKind('duration')
// A boolean-kind param never uses a capture group — its value is just whether the regex matched. }
await declareParam({
name, function resetForm() {
message_match_regex: regex, setEditingName(null)
capture_group: isBoolean || effectiveSource === 'json_field' ? '' : captureGroup,
json_field_path: effectiveSource === 'json_field' ? jsonFieldPath : null,
kind,
})
setName('') setName('')
setRegex('') setRegex('')
setCaptureGroup('') setCaptureGroup('')
setJsonFieldPath('') setJsonFieldPath('')
setSource('regex') setSource('regex')
setKind(EMPTY_KIND) setKind(EMPTY_KIND)
setSubmitError(null)
}
// Loads `param`'s current values into the declaration form for editing — a click handler for a
// declared-param badge.
function editParam(param: ParamDeclaration) {
setEditingName(param.name)
setName(param.name)
setRegex(param.message_match_regex)
setCaptureGroup(param.capture_group)
setJsonFieldPath(param.json_field_path ?? '')
setSource(param.elapsed_since_last ? 'elapsed_since_last' : param.json_field_path ? 'json_field' : 'regex')
setKind(param.kind)
setSubmitError(null)
}
async function handleSubmit(e: React.FormEvent) {
e.preventDefault()
setSubmitError(null)
setSubmitting(true)
try {
// A boolean-kind param never uses a capture group — its value is just whether the regex
// matched. An elapsed-since-last param never uses a capture group, JSON field path, or even
// its own message match regex — its value is computed, not extracted from text at all.
await declareParam({
name,
message_match_regex: effectiveSource === 'elapsed_since_last' ? '' : regex,
capture_group: isBoolean || effectiveSource !== 'regex' ? '' : captureGroup,
json_field_path: effectiveSource === 'json_field' ? jsonFieldPath : null,
kind: effectiveSource === 'elapsed_since_last' ? 'duration' : kind,
elapsed_since_last: effectiveSource === 'elapsed_since_last',
})
resetForm()
onDeclared() onDeclared()
} catch (err) { } catch (err) {
setSubmitError(err instanceof ApiError ? err.message : 'Failed to declare param') setSubmitError(err instanceof ApiError ? err.message : `Failed to ${editingName ? 'update' : 'declare'} param`)
} finally { } finally {
setSubmitting(false) setSubmitting(false)
} }
} }
// Clears every declared param (manual and auto-derived) in one action, then immediately
// re-derives whatever the engine can still find in retained log text (ticket 052) — a single
// confirmed click rather than removing params one at a time. `logs_considered` in the response
// is surfaced in the status line so it's clear the re-extraction only covers logs the engine
// still holds, not ones already evicted by a max-logs cap.
async function handleClearAll() {
if (params.length === 0) return
if (!window.confirm(`Clear all ${params.length} declared param(s)? Params will immediately be re-derived from log lines still held in memory; lines already evicted by a log cap can't regain their params.`)) {
return
}
setClearing(true)
setClearError(null)
setClearResult(null)
try {
const result = await clearParams()
setClearResult(
`Cleared ${result.removed}, re-derived ${result.reextracted} from ${result.logs_considered} retained log line(s).`,
)
resetForm()
onDeclared()
} catch (err) {
setClearError(err instanceof ApiError ? err.message : 'Failed to clear params')
} finally {
setClearing(false)
}
}
return ( return (
<div className="border-b border-border text-sm"> <div className="border-b border-border text-sm">
<div <div
@@ -95,15 +175,19 @@ export function Params({ params, onDeclared }: { params: ParamDeclaration[]; onD
<div className="flex gap-4 p-3" hidden={collapsed}> <div className="flex gap-4 p-3" hidden={collapsed}>
<form onSubmit={handleSubmit} className="w-80 shrink-0 space-y-2"> <form onSubmit={handleSubmit} className="w-80 shrink-0 space-y-2">
<div> <div>
<label className="mb-1 block text-xs text-text-subtle">Name</label> <label className="mb-1 block text-xs text-text-subtle">
Name{editingName && ' (locked while editing)'}
</label>
<input <input
className="w-full rounded border border-border-soft bg-surface-1 px-2 py-1 text-sm text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none" className="w-full rounded border border-border-soft bg-surface-1 px-2 py-1 text-sm text-text placeholder:text-text-subtle focus:border-border-strong focus:outline-none disabled:opacity-50"
value={name} value={name}
onChange={(e) => setName(e.target.value)} onChange={(e) => setName(e.target.value)}
placeholder="path" placeholder="path"
required required
disabled={editingName !== null}
/> />
</div> </div>
{!isElapsedSinceLast && (
<div> <div>
<label className="mb-1 flex items-center gap-1 text-xs text-text-subtle"> <label className="mb-1 flex items-center gap-1 text-xs text-text-subtle">
Message match regex Message match regex
@@ -117,20 +201,25 @@ export function Params({ params, onDeclared }: { params: ParamDeclaration[]; onD
required required
/> />
</div> </div>
{detectedJson && !isBoolean && ( )}
<div> <div>
<label className="mb-1 block text-xs text-text-subtle">Extraction</label> <label className="mb-1 block text-xs text-text-subtle">Extraction</label>
<select <select
className="w-full rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text focus:border-border-strong focus:outline-none" className="w-full rounded border border-border-soft bg-surface-1 px-2 py-1 text-xs text-text focus:border-border-strong focus:outline-none"
value={source} value={source}
onChange={(e) => setSource(e.target.value as ExtractionSource)} onChange={(e) => handleSourceChange(e.target.value as ExtractionSource)}
> >
<option value="regex">Regex capture group</option> <option value="regex">Regex capture group</option>
<option value="json_field">JSON field</option> {detectedJson && !isBoolean && <option value="json_field">JSON field</option>}
<option value="elapsed_since_last">Elapsed since previous event</option>
</select> </select>
</div> </div>
)} {isElapsedSinceLast ? (
{effectiveSource === 'json_field' ? ( <p className="text-xs text-text-faint">
Computed automatically — elapsed time since the previous event that shares the same
labels wherever this param is used, not parsed from the log line.
</p>
) : effectiveSource === 'json_field' ? (
<div className="flex items-end gap-1"> <div className="flex items-end gap-1">
<div className="min-w-0 flex-1"> <div className="min-w-0 flex-1">
<label className="mb-1 block text-xs text-text-subtle">JSON field path</label> <label className="mb-1 block text-xs text-text-subtle">JSON field path</label>
@@ -181,34 +270,96 @@ export function Params({ params, onDeclared }: { params: ParamDeclaration[]; onD
</select> </select>
</div> </div>
)} )}
{editingName && (
<div className="rounded border border-accent bg-accent/5 px-2 py-1 text-xs text-accent-text">
Editing "{editingName}"
</div>
)}
{submitError && <div className="rounded border border-danger-border bg-danger-bg/50 px-2 py-1 text-xs text-danger-text">{submitError}</div>} {submitError && <div className="rounded border border-danger-border bg-danger-bg/50 px-2 py-1 text-xs text-danger-text">{submitError}</div>}
<div className="flex items-center gap-2">
<button <button
type="submit" type="submit"
disabled={submitting} disabled={submitting}
className="rounded border border-border-soft px-3 py-1 text-sm text-text-secondary hover:bg-surface-2 disabled:opacity-50" className="rounded border border-border-soft px-3 py-1 text-sm text-text-secondary hover:bg-surface-2 disabled:opacity-50"
> >
{submitting ? 'Declaring…' : 'Declare param'} {submitting ? (editingName ? 'Saving…' : 'Declaring…') : editingName ? 'Save changes' : 'Declare param'}
</button> </button>
{editingName && (
<button
type="button"
onClick={resetForm}
disabled={submitting}
className="rounded border border-border-soft px-3 py-1 text-sm text-text-secondary hover:bg-surface-2 disabled:opacity-50"
>
Cancel
</button>
)}
</div>
</form> </form>
<div className="min-w-0 flex-1"> <div className="min-w-0 flex-1">
<div className="mb-1 text-xs text-text-subtle">Declared params</div> <div className="mb-1 flex items-center justify-between gap-2">
<span className="text-xs text-text-subtle">Declared params</span>
<button
type="button"
onClick={handleClearAll}
disabled={clearing || params.length === 0}
className="rounded border border-border-soft px-2 py-0.5 text-xs text-text-secondary hover:bg-surface-2 disabled:opacity-50"
title="Clear every declared param, then re-derive params from log lines still held in memory"
>
{clearing ? 'Clearing…' : 'Clear all'}
</button>
</div>
{clearResult && <div className="mb-1 text-xs text-text-subtle">{clearResult}</div>}
{clearError && <div className="mb-1 text-xs text-danger-text">{clearError}</div>}
<div className="flex flex-wrap gap-1"> <div className="flex flex-wrap gap-1">
{params.length === 0 && <span className="text-xs text-text-faint">No params declared yet.</span>} {declaredParams.length === 0 && (
{params.map((p) => ( <span className="text-xs text-text-faint">No params declared yet.</span>
<span )}
{declaredParams.map((p) => (
<button
key={p.name} key={p.name}
className="rounded border border-border-soft px-2 py-1 text-xs text-text-secondary" type="button"
title={p.message_match_regex} onClick={() => editParam(p)}
className={`rounded border px-2 py-1 text-xs ${
editingName === p.name
? 'border-accent bg-accent/10 text-accent-text'
: 'border-border-soft text-text-secondary hover:bg-surface-2'
}`}
title={p.elapsed_since_last ? undefined : p.message_match_regex}
> >
{p.name} ({p.kind} {p.name} ({p.kind}
{p.kind === 'boolean' {p.elapsed_since_last
? ', elapsed since previous'
: p.kind === 'boolean'
? '' ? ''
: `, ${p.json_field_path ? `json field ${p.json_field_path}` : `group ${p.capture_group}`}`} : p.json_field_path
? `, json field ${p.json_field_path}`
: p.capture_group !== p.name
? `, group ${p.capture_group}`
: ''}
) )
</button>
))}
</div>
{reservedParams.length > 0 && (
<>
<div className="mb-1 mt-2 text-xs text-text-subtle">
Reserved (built in — present on every line, not editable)
</div>
<div className="flex flex-wrap gap-1">
{reservedParams.map((p) => (
<span
key={p.name}
className="rounded border border-dashed border-border-soft px-2 py-1 text-xs text-text-faint"
title={`${p.name} (${p.kind}, reserved)`}
>
{p.name} ({p.kind})
</span> </span>
))} ))}
</div> </div>
</>
)}
</div> </div>
</div> </div>
</div> </div>
webapp/src/index.css
+221
View File
@@ -34,6 +34,14 @@
* composite K = H+150 (mod 360), then fill --c-surface-, --c-border-, * composite K = H+150 (mod 360), then fill --c-surface-, --c-border-,
* --c-text- as hsl(H, ...) at the saturation/lightness steps below, accent * --c-text- as hsl(H, ...) at the saturation/lightness steps below, accent
* + accent-text as hsl(C, ...), and accent-soft as hsl(K, ...). * + accent-text as hsl(C, ...), and accent-soft as hsl(K, ...).
*
* --c-accent (base look, unchanged above) is joined by two more accent
* colours (ticket 048) so charts/highlights can draw from a 3-colour set
* instead of one hue: --c-accent-2 sits at H+90 and --c-accent-3 at H+270 —
* a quadrature pair straddling the base hue, distinct from both the
* complement (accent) and the split-complement (accent-soft). These are
* purely additive: nothing that already reads --c-accent/-soft/-text
* changes look.
*/ */
:root { :root {
color-scheme: light dark; color-scheme: light dark;
@@ -62,6 +70,8 @@
--c-accent: #0ea5e9; --c-accent: #0ea5e9;
--c-accent-soft: #0369a1; --c-accent-soft: #0369a1;
--c-accent-text: #7dd3fc; --c-accent-text: #7dd3fc;
--c-accent-2: #a78bfa;
--c-accent-3: #4ade80;
} }
/* Exact lightness mirror of :root's neutral-950..50 scale (Tailwind neutral, /* Exact lightness mirror of :root's neutral-950..50 scale (Tailwind neutral,
@@ -91,6 +101,8 @@ html.light {
--c-accent: #0284c7; --c-accent: #0284c7;
--c-accent-soft: #7dd3fc; --c-accent-soft: #7dd3fc;
--c-accent-text: #0369a1; --c-accent-text: #0369a1;
--c-accent-2: #7c3aed;
--c-accent-3: #16a34a;
} }
/* base hue 0 (red): surfaces/borders/text tinted red; accent complement 180 /* base hue 0 (red): surfaces/borders/text tinted red; accent complement 180
@@ -115,6 +127,8 @@ html.red {
--c-accent: hsl(180 70% 62%); --c-accent: hsl(180 70% 62%);
--c-accent-soft: hsl(150 55% 34%); --c-accent-soft: hsl(150 55% 34%);
--c-accent-text: hsl(180 75% 80%); --c-accent-text: hsl(180 75% 80%);
--c-accent-2: hsl(90 65% 58%);
--c-accent-3: hsl(270 65% 68%);
} }
/* base hue 140 (green): surfaces/borders/text tinted green; accent complement /* base hue 140 (green): surfaces/borders/text tinted green; accent complement
@@ -139,6 +153,8 @@ html.green {
--c-accent: hsl(320 70% 62%); --c-accent: hsl(320 70% 62%);
--c-accent-soft: hsl(290 55% 34%); --c-accent-soft: hsl(290 55% 34%);
--c-accent-text: hsl(320 75% 80%); --c-accent-text: hsl(320 75% 80%);
--c-accent-2: hsl(230 65% 62%);
--c-accent-3: hsl(50 65% 58%);
} }
/* base hue 210 (blue): surfaces/borders/text tinted blue; accent complement /* base hue 210 (blue): surfaces/borders/text tinted blue; accent complement
@@ -163,6 +179,8 @@ html.blue {
--c-accent: hsl(30 70% 62%); --c-accent: hsl(30 70% 62%);
--c-accent-soft: hsl(0 55% 34%); --c-accent-soft: hsl(0 55% 34%);
--c-accent-text: hsl(30 75% 80%); --c-accent-text: hsl(30 75% 80%);
--c-accent-2: hsl(300 65% 68%);
--c-accent-3: hsl(120 65% 55%);
} }
/* base hue 50 (yellow): surfaces/borders/text tinted yellow; accent /* base hue 50 (yellow): surfaces/borders/text tinted yellow; accent
@@ -187,6 +205,8 @@ html.yellow {
--c-accent: hsl(230 70% 62%); --c-accent: hsl(230 70% 62%);
--c-accent-soft: hsl(200 55% 34%); --c-accent-soft: hsl(200 55% 34%);
--c-accent-text: hsl(230 75% 80%); --c-accent-text: hsl(230 75% 80%);
--c-accent-2: hsl(140 65% 55%);
--c-accent-3: hsl(320 65% 68%);
} }
/* base hue 330 (pink): surfaces/borders/text tinted pink; accent complement /* base hue 330 (pink): surfaces/borders/text tinted pink; accent complement
@@ -211,6 +231,8 @@ html.pink {
--c-accent: hsl(150 70% 62%); --c-accent: hsl(150 70% 62%);
--c-accent-soft: hsl(120 55% 34%); --c-accent-soft: hsl(120 55% 34%);
--c-accent-text: hsl(150 75% 80%); --c-accent-text: hsl(150 75% 80%);
--c-accent-2: hsl(60 65% 58%);
--c-accent-3: hsl(240 65% 68%);
} }
/* base hue 30 (orange): surfaces/borders/text tinted orange; accent /* base hue 30 (orange): surfaces/borders/text tinted orange; accent
@@ -235,6 +257,203 @@ html.orange {
--c-accent: hsl(210 70% 62%); --c-accent: hsl(210 70% 62%);
--c-accent-soft: hsl(180 55% 34%); --c-accent-soft: hsl(180 55% 34%);
--c-accent-text: hsl(210 75% 80%); --c-accent-text: hsl(210 75% 80%);
--c-accent-2: hsl(120 65% 55%);
--c-accent-3: hsl(300 65% 68%);
}
/* Brighter, light-surfaced counterpart to `green` (ticket 048): same base
hue (140) and accent hues as `green`, but with a `light`-style inverted
surface/border/text scale (high lightness instead of low) so it reads as
a distinctly brighter theme, not a re-tint of the dark `green`. Danger
colours are `light`'s (hue-neutral), matching how `light` itself doesn't
tint danger either. */
html.light_green {
--c-surface-0: hsl(140 40% 96%);
--c-surface-1: hsl(140 35% 92%);
--c-surface-2: hsl(140 30% 85%);
--c-border: hsl(140 25% 80%);
--c-border-soft: hsl(140 22% 70%);
--c-border-faint: hsl(140 35% 92%);
--c-border-strong: hsl(140 20% 40%);
--c-text: hsl(140 30% 12%);
--c-text-secondary: hsl(140 20% 25%);
--c-text-muted: hsl(140 15% 40%);
--c-text-subtle: hsl(140 10% 50%);
--c-text-faint: hsl(140 10% 65%);
--c-text-faintest: hsl(140 12% 80%);
--c-danger-bg: #fee2e2;
--c-danger-border: #fca5a5;
--c-danger-text: #b91c1c;
--c-danger-text-strong: #991b1b;
--c-accent: hsl(320 70% 40%);
--c-accent-soft: hsl(290 60% 85%);
--c-accent-text: hsl(320 75% 28%);
--c-accent-2: hsl(230 70% 45%);
--c-accent-3: hsl(50 70% 38%);
}
/* Brighter, light-surfaced counterpart to `red` (ticket 056): same base hue
(0) and accent hues as `red`, with `light`'s inverted surface/border/text
scale and `light`'s hue-neutral danger colours. */
html.light_red {
--c-surface-0: hsl(0 40% 96%);
--c-surface-1: hsl(0 35% 92%);
--c-surface-2: hsl(0 30% 85%);
--c-border: hsl(0 25% 80%);
--c-border-soft: hsl(0 22% 70%);
--c-border-faint: hsl(0 35% 92%);
--c-border-strong: hsl(0 20% 40%);
--c-text: hsl(0 30% 12%);
--c-text-secondary: hsl(0 20% 25%);
--c-text-muted: hsl(0 15% 40%);
--c-text-subtle: hsl(0 10% 50%);
--c-text-faint: hsl(0 10% 65%);
--c-text-faintest: hsl(0 12% 80%);
--c-danger-bg: #fee2e2;
--c-danger-border: #fca5a5;
--c-danger-text: #b91c1c;
--c-danger-text-strong: #991b1b;
--c-accent: hsl(180 70% 40%);
--c-accent-soft: hsl(150 60% 85%);
--c-accent-text: hsl(180 75% 28%);
--c-accent-2: hsl(90 70% 45%);
--c-accent-3: hsl(270 70% 38%);
}
/* Brighter, light-surfaced counterpart to `blue` (ticket 056): same base hue
(210) and accent hues as `blue`, with `light`'s inverted surface/border/text
scale and `light`'s hue-neutral danger colours. */
html.light_blue {
--c-surface-0: hsl(210 40% 96%);
--c-surface-1: hsl(210 35% 92%);
--c-surface-2: hsl(210 30% 85%);
--c-border: hsl(210 25% 80%);
--c-border-soft: hsl(210 22% 70%);
--c-border-faint: hsl(210 35% 92%);
--c-border-strong: hsl(210 20% 40%);
--c-text: hsl(210 30% 12%);
--c-text-secondary: hsl(210 20% 25%);
--c-text-muted: hsl(210 15% 40%);
--c-text-subtle: hsl(210 10% 50%);
--c-text-faint: hsl(210 10% 65%);
--c-text-faintest: hsl(210 12% 80%);
--c-danger-bg: #fee2e2;
--c-danger-border: #fca5a5;
--c-danger-text: #b91c1c;
--c-danger-text-strong: #991b1b;
--c-accent: hsl(30 70% 40%);
--c-accent-soft: hsl(0 60% 85%);
--c-accent-text: hsl(30 75% 28%);
--c-accent-2: hsl(300 70% 45%);
--c-accent-3: hsl(120 70% 38%);
}
/* Brighter, light-surfaced counterpart to `yellow` (ticket 056): same base
hue (50) and accent hues as `yellow`, with `light`'s inverted
surface/border/text scale and `light`'s hue-neutral danger colours. */
html.light_yellow {
--c-surface-0: hsl(50 40% 96%);
--c-surface-1: hsl(50 35% 92%);
--c-surface-2: hsl(50 30% 85%);
--c-border: hsl(50 25% 80%);
--c-border-soft: hsl(50 22% 70%);
--c-border-faint: hsl(50 35% 92%);
--c-border-strong: hsl(50 20% 40%);
--c-text: hsl(50 30% 12%);
--c-text-secondary: hsl(50 20% 25%);
--c-text-muted: hsl(50 15% 40%);
--c-text-subtle: hsl(50 10% 50%);
--c-text-faint: hsl(50 10% 65%);
--c-text-faintest: hsl(50 12% 80%);
--c-danger-bg: #fee2e2;
--c-danger-border: #fca5a5;
--c-danger-text: #b91c1c;
--c-danger-text-strong: #991b1b;
--c-accent: hsl(230 70% 40%);
--c-accent-soft: hsl(200 60% 85%);
--c-accent-text: hsl(230 75% 28%);
--c-accent-2: hsl(140 70% 45%);
--c-accent-3: hsl(320 70% 38%);
}
/* Brighter, light-surfaced counterpart to `pink` (ticket 056): same base hue
(330) and accent hues as `pink`, with `light`'s inverted surface/border/text
scale and `light`'s hue-neutral danger colours. */
html.light_pink {
--c-surface-0: hsl(330 40% 96%);
--c-surface-1: hsl(330 35% 92%);
--c-surface-2: hsl(330 30% 85%);
--c-border: hsl(330 25% 80%);
--c-border-soft: hsl(330 22% 70%);
--c-border-faint: hsl(330 35% 92%);
--c-border-strong: hsl(330 20% 40%);
--c-text: hsl(330 30% 12%);
--c-text-secondary: hsl(330 20% 25%);
--c-text-muted: hsl(330 15% 40%);
--c-text-subtle: hsl(330 10% 50%);
--c-text-faint: hsl(330 10% 65%);
--c-text-faintest: hsl(330 12% 80%);
--c-danger-bg: #fee2e2;
--c-danger-border: #fca5a5;
--c-danger-text: #b91c1c;
--c-danger-text-strong: #991b1b;
--c-accent: hsl(150 70% 40%);
--c-accent-soft: hsl(120 60% 85%);
--c-accent-text: hsl(150 75% 28%);
--c-accent-2: hsl(60 70% 45%);
--c-accent-3: hsl(240 70% 38%);
}
/* Brighter, light-surfaced counterpart to `orange` (ticket 056): same base
hue (30) and accent hues as `orange`, with `light`'s inverted
surface/border/text scale and `light`'s hue-neutral danger colours. */
html.light_orange {
--c-surface-0: hsl(30 40% 96%);
--c-surface-1: hsl(30 35% 92%);
--c-surface-2: hsl(30 30% 85%);
--c-border: hsl(30 25% 80%);
--c-border-soft: hsl(30 22% 70%);
--c-border-faint: hsl(30 35% 92%);
--c-border-strong: hsl(30 20% 40%);
--c-text: hsl(30 30% 12%);
--c-text-secondary: hsl(30 20% 25%);
--c-text-muted: hsl(30 15% 40%);
--c-text-subtle: hsl(30 10% 50%);
--c-text-faint: hsl(30 10% 65%);
--c-text-faintest: hsl(30 12% 80%);
--c-danger-bg: #fee2e2;
--c-danger-border: #fca5a5;
--c-danger-text: #b91c1c;
--c-danger-text-strong: #991b1b;
--c-accent: hsl(210 70% 40%);
--c-accent-soft: hsl(180 60% 85%);
--c-accent-text: hsl(210 75% 28%);
--c-accent-2: hsl(120 70% 45%);
--c-accent-3: hsl(300 70% 38%);
} }
@theme inline { @theme inline {
@@ -262,6 +481,8 @@ html.orange {
--color-accent: var(--c-accent); --color-accent: var(--c-accent);
--color-accent-soft: var(--c-accent-soft); --color-accent-soft: var(--c-accent-soft);
--color-accent-text: var(--c-accent-text); --color-accent-text: var(--c-accent-text);
--color-accent-2: var(--c-accent-2);
--color-accent-3: var(--c-accent-3);
} }
html, html,
webapp/src/lib/api.ts
+41 -13
View File
@@ -30,6 +30,10 @@ export interface ParamDeclaration {
/** Alternative to `capture_group`: a dot-separated JSON field path (e.g. "foo.bar"), read from lines that parse as a JSON object. When set (non-empty), takes precedence over `capture_group`. */ /** Alternative to `capture_group`: a dot-separated JSON field path (e.g. "foo.bar"), read from lines that parse as a JSON object. When set (non-empty), takes precedence over `capture_group`. */
json_field_path?: string | null json_field_path?: string | null
kind: 'string' | 'number' | 'timestamp' | 'duration' | 'boolean' kind: 'string' | 'number' | 'timestamp' | 'duration' | 'boolean'
/** A third extraction source, distinct from `capture_group`/`json_field_path`: when true, this param's value is never parsed from text — it's computed as elapsed time since the previous event sharing the same label set. Only meaningful paired with `kind: 'duration'`. */
elapsed_since_last?: boolean
/** Set only for the built-in `ottyl_seq`/`ottyl_ts`/`ottyl_fmt` names, synthesized into `GET /api/params` for discoverability — they're usable in the filter bar (and `ottyl_seq`/`ottyl_ts` in dashboard PromQL queries) without ever being declared, and can't be edited or removed. */
reserved?: boolean
} }
export interface TimelineBucket { export interface TimelineBucket {
@@ -45,24 +49,17 @@ export interface Stats {
last_timestamp_ms: number | null last_timestamp_ms: number | null
/** Ticket 011: running approximate byte size of all currently-ingested logs. */ /** Ticket 011: running approximate byte size of all currently-ingested logs. */
approx_bytes_total: number approx_bytes_total: number
} /** Ticket 073: highest ingestion-to-engine-processing lag (ms) seen within the trailing 5s. */
ingest_lag_high_watermark_ms: number
export type ParamRole = 'label' | 'value'
export interface MetricParamMatcher {
param_name: string
capture_group: string
/** See `ParamDeclaration.json_field_path`. */
json_field_path?: string | null
kind: 'string' | 'number' | 'timestamp' | 'duration'
/** Defaults to 'label' server-side when omitted. */
role?: ParamRole
} }
export interface MetricDeclaration { export interface MetricDeclaration {
name: string name: string
message_match_regex: string message_match_regex: string
params: MetricParamMatcher[] /** Names of already-declared `ParamDeclaration`s (see the Params panel) whose values become this metric's bucket labels. */
label_params: string[]
/** Name of an already-declared param (if any) whose value becomes this metric's per-event bucket value instead of a label. */
value_param?: string | null
/** Fixed labels applied on a match/non-match, on top of any captured params. */ /** Fixed labels applied on a match/non-match, on top of any captured params. */
matched_labels?: Record<string, string> | null matched_labels?: Record<string, string> | null
unmatched_labels?: Record<string, string> | null unmatched_labels?: Record<string, string> | null
@@ -97,6 +94,12 @@ export type ColorScheme =
| 'yellow' | 'yellow'
| 'pink' | 'pink'
| 'orange' | 'orange'
| 'light_green'
| 'light_red'
| 'light_blue'
| 'light_yellow'
| 'light_pink'
| 'light_orange'
export interface Settings { export interface Settings {
color_scheme: ColorScheme color_scheme: ColorScheme
@@ -251,6 +254,10 @@ export function fetchLogs(params: {
endMs?: number endMs?: number
limit?: number limit?: number
filter?: string filter?: string
/** Ticket 045: an already-declared param name to order entries by, instead of the default (`ottyl_ts`-tracking) order. */
sortBy?: string
/** Ticket 045: only meaningful alongside `sortBy`. `true` (default) orders lines missing that param by `ottyl_seq`, appended after the sorted lines; `false` drops them. */
sortMissingFallback?: boolean
}): Promise<LogEntry[]> { }): Promise<LogEntry[]> {
const search = new URLSearchParams() const search = new URLSearchParams()
if (params.sinceSequence !== undefined) search.set('since_sequence', String(params.sinceSequence)) if (params.sinceSequence !== undefined) search.set('since_sequence', String(params.sinceSequence))
@@ -258,6 +265,8 @@ export function fetchLogs(params: {
if (params.endMs !== undefined) search.set('end_ms', String(params.endMs)) if (params.endMs !== undefined) search.set('end_ms', String(params.endMs))
if (params.limit !== undefined) search.set('limit', String(params.limit)) if (params.limit !== undefined) search.set('limit', String(params.limit))
if (params.filter) search.set('filter', params.filter) if (params.filter) search.set('filter', params.filter)
if (params.sortBy) search.set('sort_by', params.sortBy)
if (params.sortMissingFallback !== undefined) search.set('sort_missing_fallback', String(params.sortMissingFallback))
return getJson(`/api/logs?${search.toString()}`) return getJson(`/api/logs?${search.toString()}`)
} }
@@ -287,6 +296,21 @@ export function declareParam(request: ParamDeclaration): Promise<ParamDeclaratio
return postJson('/api/params', request) return postJson('/api/params', request)
} }
export interface ClearParamsResult {
/** How many param declarations (manual + auto-derived) were removed. */
removed: number
/** How many params were immediately re-derived from log text still retained by the engine. */
reextracted: number
/** How many retained log lines were re-scanned — lines already evicted by `max_logs` before the clear can't regain their params, so this can be smaller than the total lines ever ingested. */
logs_considered: number
}
// Clears every declared param (ticket 052) and re-derives whatever the engine's still-retained log
// text supports — see `ClearParamsResult`'s fields for what came back.
export function clearParams(): Promise<ClearParamsResult> {
return sendJson('/api/params', 'DELETE')
}
export function fetchMetricDeclarations(): Promise<MetricDeclaration[]> { export function fetchMetricDeclarations(): Promise<MetricDeclaration[]> {
return getJson('/api/metrics') return getJson('/api/metrics')
} }
@@ -300,6 +324,10 @@ export function updateMetric(existingName: string, request: MetricDeclaration):
return putJson(`/api/metrics/${encodeURIComponent(existingName)}`, request) return putJson(`/api/metrics/${encodeURIComponent(existingName)}`, request)
} }
export function deleteMetric(name: string): Promise<void> {
return sendJson(`/api/metrics/${encodeURIComponent(name)}`, 'DELETE')
}
export function fetchMetricBucketValues( export function fetchMetricBucketValues(
name: string, name: string,
params: { params: {
webapp/src/lib/themes.ts
+169 -1
View File
@@ -1,4 +1,10 @@
// A theme file is a CSS file with one `:root { --c-*: ...; }` block defining every variable in `THEME_CSS_VARS`; `useSettingsPanel` applies the selected one as an unscoped stylesheet only while no built-in `html.<scheme>` class is present, so it wins the cascade without rewriting the selector. // A user-supplied style is any CSS file that defines every variable in `THEME_CSS_VARS` somewhere in
// a `:root { --c-*: ...; }` block (ticket 020/047: styles are selectable on equal footing with the
// built-ins, and there's no requirement that this block be the *only* thing the file contains — a
// style is free to also target any other selector, e.g. to restyle layout, spacing, or fonts, not
// just the palette). `useSettingsPanel` applies the selected one as an unscoped stylesheet only
// while no built-in `html.<scheme>` class is present, so it wins the cascade without rewriting the
// selector.
/** Canonical list of every `--c-*` variable every theme in `index.css` defines. */ /** Canonical list of every `--c-*` variable every theme in `index.css` defines. */
export const THEME_CSS_VARS = [ export const THEME_CSS_VARS = [
@@ -22,6 +28,8 @@ export const THEME_CSS_VARS = [
'--c-accent', '--c-accent',
'--c-accent-soft', '--c-accent-soft',
'--c-accent-text', '--c-accent-text',
'--c-accent-2',
'--c-accent-3',
] as const ] as const
export type ThemeVars = Record<(typeof THEME_CSS_VARS)[number], string> export type ThemeVars = Record<(typeof THEME_CSS_VARS)[number], string>
@@ -49,6 +57,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': '#0ea5e9', '--c-accent': '#0ea5e9',
'--c-accent-soft': '#0369a1', '--c-accent-soft': '#0369a1',
'--c-accent-text': '#7dd3fc', '--c-accent-text': '#7dd3fc',
'--c-accent-2': '#a78bfa',
'--c-accent-3': '#4ade80',
}, },
light: { light: {
'--c-surface-0': '#fafafa', '--c-surface-0': '#fafafa',
@@ -71,6 +81,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': '#0284c7', '--c-accent': '#0284c7',
'--c-accent-soft': '#7dd3fc', '--c-accent-soft': '#7dd3fc',
'--c-accent-text': '#0369a1', '--c-accent-text': '#0369a1',
'--c-accent-2': '#7c3aed',
'--c-accent-3': '#16a34a',
}, },
red: { red: {
'--c-surface-0': 'hsl(0 32% 10%)', '--c-surface-0': 'hsl(0 32% 10%)',
@@ -93,6 +105,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': 'hsl(180 70% 62%)', '--c-accent': 'hsl(180 70% 62%)',
'--c-accent-soft': 'hsl(150 55% 34%)', '--c-accent-soft': 'hsl(150 55% 34%)',
'--c-accent-text': 'hsl(180 75% 80%)', '--c-accent-text': 'hsl(180 75% 80%)',
'--c-accent-2': 'hsl(90 65% 58%)',
'--c-accent-3': 'hsl(270 65% 68%)',
}, },
green: { green: {
'--c-surface-0': 'hsl(140 32% 10%)', '--c-surface-0': 'hsl(140 32% 10%)',
@@ -115,6 +129,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': 'hsl(320 70% 62%)', '--c-accent': 'hsl(320 70% 62%)',
'--c-accent-soft': 'hsl(290 55% 34%)', '--c-accent-soft': 'hsl(290 55% 34%)',
'--c-accent-text': 'hsl(320 75% 80%)', '--c-accent-text': 'hsl(320 75% 80%)',
'--c-accent-2': 'hsl(230 65% 62%)',
'--c-accent-3': 'hsl(50 65% 58%)',
}, },
blue: { blue: {
'--c-surface-0': 'hsl(210 32% 10%)', '--c-surface-0': 'hsl(210 32% 10%)',
@@ -137,6 +153,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': 'hsl(30 70% 62%)', '--c-accent': 'hsl(30 70% 62%)',
'--c-accent-soft': 'hsl(0 55% 34%)', '--c-accent-soft': 'hsl(0 55% 34%)',
'--c-accent-text': 'hsl(30 75% 80%)', '--c-accent-text': 'hsl(30 75% 80%)',
'--c-accent-2': 'hsl(300 65% 68%)',
'--c-accent-3': 'hsl(120 65% 55%)',
}, },
yellow: { yellow: {
'--c-surface-0': 'hsl(50 32% 10%)', '--c-surface-0': 'hsl(50 32% 10%)',
@@ -159,6 +177,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': 'hsl(230 70% 62%)', '--c-accent': 'hsl(230 70% 62%)',
'--c-accent-soft': 'hsl(200 55% 34%)', '--c-accent-soft': 'hsl(200 55% 34%)',
'--c-accent-text': 'hsl(230 75% 80%)', '--c-accent-text': 'hsl(230 75% 80%)',
'--c-accent-2': 'hsl(140 65% 55%)',
'--c-accent-3': 'hsl(320 65% 68%)',
}, },
pink: { pink: {
'--c-surface-0': 'hsl(330 32% 10%)', '--c-surface-0': 'hsl(330 32% 10%)',
@@ -181,6 +201,8 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': 'hsl(150 70% 62%)', '--c-accent': 'hsl(150 70% 62%)',
'--c-accent-soft': 'hsl(120 55% 34%)', '--c-accent-soft': 'hsl(120 55% 34%)',
'--c-accent-text': 'hsl(150 75% 80%)', '--c-accent-text': 'hsl(150 75% 80%)',
'--c-accent-2': 'hsl(60 65% 58%)',
'--c-accent-3': 'hsl(240 65% 68%)',
}, },
orange: { orange: {
'--c-surface-0': 'hsl(30 32% 10%)', '--c-surface-0': 'hsl(30 32% 10%)',
@@ -203,6 +225,152 @@ export const BUILTIN_THEMES: Record<string, ThemeVars> = {
'--c-accent': 'hsl(210 70% 62%)', '--c-accent': 'hsl(210 70% 62%)',
'--c-accent-soft': 'hsl(180 55% 34%)', '--c-accent-soft': 'hsl(180 55% 34%)',
'--c-accent-text': 'hsl(210 75% 80%)', '--c-accent-text': 'hsl(210 75% 80%)',
'--c-accent-2': 'hsl(120 65% 55%)',
'--c-accent-3': 'hsl(300 65% 68%)',
},
light_green: {
'--c-surface-0': 'hsl(140 40% 96%)',
'--c-surface-1': 'hsl(140 35% 92%)',
'--c-surface-2': 'hsl(140 30% 85%)',
'--c-border': 'hsl(140 25% 80%)',
'--c-border-soft': 'hsl(140 22% 70%)',
'--c-border-faint': 'hsl(140 35% 92%)',
'--c-border-strong': 'hsl(140 20% 40%)',
'--c-text': 'hsl(140 30% 12%)',
'--c-text-secondary': 'hsl(140 20% 25%)',
'--c-text-muted': 'hsl(140 15% 40%)',
'--c-text-subtle': 'hsl(140 10% 50%)',
'--c-text-faint': 'hsl(140 10% 65%)',
'--c-text-faintest': 'hsl(140 12% 80%)',
'--c-danger-bg': '#fee2e2',
'--c-danger-border': '#fca5a5',
'--c-danger-text': '#b91c1c',
'--c-danger-text-strong': '#991b1b',
'--c-accent': 'hsl(320 70% 40%)',
'--c-accent-soft': 'hsl(290 60% 85%)',
'--c-accent-text': 'hsl(320 75% 28%)',
'--c-accent-2': 'hsl(230 70% 45%)',
'--c-accent-3': 'hsl(50 70% 38%)',
},
light_red: {
'--c-surface-0': 'hsl(0 40% 96%)',
'--c-surface-1': 'hsl(0 35% 92%)',
'--c-surface-2': 'hsl(0 30% 85%)',
'--c-border': 'hsl(0 25% 80%)',
'--c-border-soft': 'hsl(0 22% 70%)',
'--c-border-faint': 'hsl(0 35% 92%)',
'--c-border-strong': 'hsl(0 20% 40%)',
'--c-text': 'hsl(0 30% 12%)',
'--c-text-secondary': 'hsl(0 20% 25%)',
'--c-text-muted': 'hsl(0 15% 40%)',
'--c-text-subtle': 'hsl(0 10% 50%)',
'--c-text-faint': 'hsl(0 10% 65%)',
'--c-text-faintest': 'hsl(0 12% 80%)',
'--c-danger-bg': '#fee2e2',
'--c-danger-border': '#fca5a5',
'--c-danger-text': '#b91c1c',
'--c-danger-text-strong': '#991b1b',
'--c-accent': 'hsl(180 70% 40%)',
'--c-accent-soft': 'hsl(150 60% 85%)',
'--c-accent-text': 'hsl(180 75% 28%)',
'--c-accent-2': 'hsl(90 70% 45%)',
'--c-accent-3': 'hsl(270 70% 38%)',
},
light_blue: {
'--c-surface-0': 'hsl(210 40% 96%)',
'--c-surface-1': 'hsl(210 35% 92%)',
'--c-surface-2': 'hsl(210 30% 85%)',
'--c-border': 'hsl(210 25% 80%)',
'--c-border-soft': 'hsl(210 22% 70%)',
'--c-border-faint': 'hsl(210 35% 92%)',
'--c-border-strong': 'hsl(210 20% 40%)',
'--c-text': 'hsl(210 30% 12%)',
'--c-text-secondary': 'hsl(210 20% 25%)',
'--c-text-muted': 'hsl(210 15% 40%)',
'--c-text-subtle': 'hsl(210 10% 50%)',
'--c-text-faint': 'hsl(210 10% 65%)',
'--c-text-faintest': 'hsl(210 12% 80%)',
'--c-danger-bg': '#fee2e2',
'--c-danger-border': '#fca5a5',
'--c-danger-text': '#b91c1c',
'--c-danger-text-strong': '#991b1b',
'--c-accent': 'hsl(30 70% 40%)',
'--c-accent-soft': 'hsl(0 60% 85%)',
'--c-accent-text': 'hsl(30 75% 28%)',
'--c-accent-2': 'hsl(300 70% 45%)',
'--c-accent-3': 'hsl(120 70% 38%)',
},
light_yellow: {
'--c-surface-0': 'hsl(50 40% 96%)',
'--c-surface-1': 'hsl(50 35% 92%)',
'--c-surface-2': 'hsl(50 30% 85%)',
'--c-border': 'hsl(50 25% 80%)',
'--c-border-soft': 'hsl(50 22% 70%)',
'--c-border-faint': 'hsl(50 35% 92%)',
'--c-border-strong': 'hsl(50 20% 40%)',
'--c-text': 'hsl(50 30% 12%)',
'--c-text-secondary': 'hsl(50 20% 25%)',
'--c-text-muted': 'hsl(50 15% 40%)',
'--c-text-subtle': 'hsl(50 10% 50%)',
'--c-text-faint': 'hsl(50 10% 65%)',
'--c-text-faintest': 'hsl(50 12% 80%)',
'--c-danger-bg': '#fee2e2',
'--c-danger-border': '#fca5a5',
'--c-danger-text': '#b91c1c',
'--c-danger-text-strong': '#991b1b',
'--c-accent': 'hsl(230 70% 40%)',
'--c-accent-soft': 'hsl(200 60% 85%)',
'--c-accent-text': 'hsl(230 75% 28%)',
'--c-accent-2': 'hsl(140 70% 45%)',
'--c-accent-3': 'hsl(320 70% 38%)',
},
light_pink: {
'--c-surface-0': 'hsl(330 40% 96%)',
'--c-surface-1': 'hsl(330 35% 92%)',
'--c-surface-2': 'hsl(330 30% 85%)',
'--c-border': 'hsl(330 25% 80%)',
'--c-border-soft': 'hsl(330 22% 70%)',
'--c-border-faint': 'hsl(330 35% 92%)',
'--c-border-strong': 'hsl(330 20% 40%)',
'--c-text': 'hsl(330 30% 12%)',
'--c-text-secondary': 'hsl(330 20% 25%)',
'--c-text-muted': 'hsl(330 15% 40%)',
'--c-text-subtle': 'hsl(330 10% 50%)',
'--c-text-faint': 'hsl(330 10% 65%)',
'--c-text-faintest': 'hsl(330 12% 80%)',
'--c-danger-bg': '#fee2e2',
'--c-danger-border': '#fca5a5',
'--c-danger-text': '#b91c1c',
'--c-danger-text-strong': '#991b1b',
'--c-accent': 'hsl(150 70% 40%)',
'--c-accent-soft': 'hsl(120 60% 85%)',
'--c-accent-text': 'hsl(150 75% 28%)',
'--c-accent-2': 'hsl(60 70% 45%)',
'--c-accent-3': 'hsl(240 70% 38%)',
},
light_orange: {
'--c-surface-0': 'hsl(30 40% 96%)',
'--c-surface-1': 'hsl(30 35% 92%)',
'--c-surface-2': 'hsl(30 30% 85%)',
'--c-border': 'hsl(30 25% 80%)',
'--c-border-soft': 'hsl(30 22% 70%)',
'--c-border-faint': 'hsl(30 35% 92%)',
'--c-border-strong': 'hsl(30 20% 40%)',
'--c-text': 'hsl(30 30% 12%)',
'--c-text-secondary': 'hsl(30 20% 25%)',
'--c-text-muted': 'hsl(30 15% 40%)',
'--c-text-subtle': 'hsl(30 10% 50%)',
'--c-text-faint': 'hsl(30 10% 65%)',
'--c-text-faintest': 'hsl(30 12% 80%)',
'--c-danger-bg': '#fee2e2',
'--c-danger-border': '#fca5a5',
'--c-danger-text': '#b91c1c',
'--c-danger-text-strong': '#991b1b',
'--c-accent': 'hsl(210 70% 40%)',
'--c-accent-soft': 'hsl(180 60% 85%)',
'--c-accent-text': 'hsl(210 75% 28%)',
'--c-accent-2': 'hsl(120 70% 45%)',
'--c-accent-3': 'hsl(300 70% 38%)',
}, },
} }
webapp/src/lib/useSettingsPanel.ts
+18 -2
View File
@@ -21,7 +21,22 @@ import {
const INJECTED_THEME_ATTR = 'data-ottyl-theme-css-id' const INJECTED_THEME_ATTR = 'data-ottyl-theme-css-id'
// Themes selectable independently of `system`, each backed by a `html.<name>` block in `index.css`. // Themes selectable independently of `system`, each backed by a `html.<name>` block in `index.css`.
const THEME_CLASSES = ['light', 'dark', 'red', 'green', 'blue', 'yellow', 'pink', 'orange'] as const const THEME_CLASSES = [
'light',
'dark',
'red',
'green',
'blue',
'yellow',
'pink',
'orange',
'light_green',
'light_red',
'light_blue',
'light_yellow',
'light_pink',
'light_orange',
] as const
// Reflects `colorScheme` onto `<html>` as a class + `data-color-scheme`; when a custom theme is active, no built-in class is added at all, since a class selector would out-specificity its injected `:root` override (see `applyCustomThemeCss`) and mask it. // Reflects `colorScheme` onto `<html>` as a class + `data-color-scheme`; when a custom theme is active, no built-in class is added at all, since a class selector would out-specificity its injected `:root` override (see `applyCustomThemeCss`) and mask it.
function applyColorScheme(colorScheme: ColorScheme, customThemeActive: boolean) { function applyColorScheme(colorScheme: ColorScheme, customThemeActive: boolean) {
@@ -143,7 +158,8 @@ export function useSettingsPanel() {
[guarded], [guarded],
) )
// Every entry is always a selectable theme (ticket 020), so there's a single import path. // Every entry is always a selectable style, on equal footing with the built-in colour schemes
// (ticket 020 unified the data model; ticket 047 unified the UI), so there's a single import path.
const importCssEntry = useCallback( const importCssEntry = useCallback(
(filename: string, content: string) => (filename: string, content: string) =>
guarded(async () => { guarded(async () => {