Resolve tickets 059-061: generic log-stream columns + sort helper indexes
- 059: log stream header/column detection is now generic whitespace-columnar splitting (2+ spaces, single tab) instead of one fixed six-column regex. Declared params come from whatever columns the header actually names, plus a trailing unlabelled message field that's never swallowed by the last named column. - 060: timeline sort-by-property now walks an incrementally-maintained per-sort-key helper index (BTreeMap keyed so "has the param, ordered by value" sorts before "missing it, ordered by ottyl_seq") instead of materializing and re-sorting the full filtered range on every query. - 061: helper indexes are swept for staleness on ingest and dropped after 5 minutes unused; a purged (or never-built) index transparently rebuilds via the same lazy one-time scan on the next sorted query. - Also invalidate a param's helper index on manual re-declaration, removal, or ClearParams, so a redefined regex can't leave a sort helper index serving values derived under the old declaration. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
This commit is contained in:
Hugh Kaznowski
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
4ace654019
commit
de9ba6f01b
@@ -1,62 +0,0 @@
|
||||
# 059 — `log stream` parsing is one hardcoded regex, not generic whitespace-columnar detection
|
||||
|
||||
## Summary
|
||||
|
||||
Current `log stream` support is a single hardcoded regex,
|
||||
`LOG_STREAM_RE` (`src/logfmt.rs:93-113`), matching exactly six fixed
|
||||
columns (`Timestamp Thread Type Activity PID TTL`) by content shape
|
||||
(dates/hex/decimals) rather than by parsing the header row generically.
|
||||
`auto_declare_log_stream_params` (`src/engine.rs:451-472`) then extracts
|
||||
only three hardcoded fields (`log_type`, `process`, `pid`) via three more
|
||||
bespoke regexes (`LOG_STREAM_TYPE_PATTERN` etc., `logfmt.rs:108-113`).
|
||||
|
||||
The fixture `src/fixtures/log_stream_sample.txt` has a real header row
|
||||
(`Timestamp Thread Type Activity PID TTL `)
|
||||
whose columns are separated by runs of 2+ spaces — the general CSV-like
|
||||
shape the wishlist asks for — plus a trailing unlabelled `process:
|
||||
message` tail that must not get merged into the preceding column. Today
|
||||
nothing derives column names or positions from that header; every field
|
||||
is matched by a fixed, hand-tuned regex, so any deviation from this
|
||||
exact `log stream` output shape silently fails to parse.
|
||||
|
||||
## Scope
|
||||
|
||||
- Replace `LOG_STREAM_RE`/`LOG_STREAM_*_PATTERN`'s fixed-shape matching
|
||||
with a generic whitespace-columnar detector: given a header line,
|
||||
split on runs of duplicated whitespace (2+ spaces, or a single tab, or
|
||||
" \t " — any whitespace repeated) to determine column names and
|
||||
positions, then apply the same column boundaries to subsequent data
|
||||
lines.
|
||||
- The header's last named column's value must still capture through
|
||||
to end-of-line without swallowing the trailing unlabelled message
|
||||
column that `log stream` appends after the last header-named field
|
||||
(i.e. that trailing text becomes its own unlabelled/message param,
|
||||
not merged into the last header column).
|
||||
- Single tabs are acceptable separators even where surrounding spaces
|
||||
aren't doubled.
|
||||
- Wire the new column-driven extraction into
|
||||
`auto_declare_log_stream_params` in place of the three bespoke
|
||||
per-field regexes, so declared params are derived from whatever
|
||||
columns the header actually names, not a fixed list of three.
|
||||
- Use `src/fixtures/log_stream_sample.txt` as the reference fixture and
|
||||
add/extend tests in `src/logfmt.rs`/`src/engine.rs` asserting header
|
||||
columns are correctly detected and the trailing message field is kept
|
||||
separate.
|
||||
- Out of scope: changing param-cardinality bounding introduced in ticket
|
||||
050/054 (commit f48d3c6f) — this is purely about how columns are
|
||||
detected/split, not about pruning noisy auto-declarations.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Piping real `log stream` output (as in `log_stream_sample.txt`)
|
||||
through ottyl declares params matching the header's actual column
|
||||
names, not the previous fixed three.
|
||||
- The trailing unlabelled message text on each `log stream` line is
|
||||
captured as its own field and never merged into the last named
|
||||
header column.
|
||||
- A header/data line whose columns are separated by a single tab is
|
||||
parsed identically to one separated by 2+ spaces.
|
||||
|
||||
## Depends on
|
||||
|
||||
None.
|
||||
@@ -1,57 +0,0 @@
|
||||
# 060 — Timeline sort-by-property does a full materialize+sort per query instead of using a helper index
|
||||
|
||||
## Summary
|
||||
|
||||
Sorting the timeline by an arbitrary property does work today (from
|
||||
ticket 045): in the `EngineQuery::Logs` branch (`src/engine.rs:1408-1487`),
|
||||
when `sort_by` is set, every filtered line is materialized into a
|
||||
`Vec<LogEntryDto>` (deriving params for all of them), then sorted
|
||||
in-memory with `.sort_by(...)`, with `sort_missing_fallback` (default
|
||||
true) partitioning entries that have the sort param from those that
|
||||
don't and ordering the latter by `ottyl_seq` (`:1468-1482`).
|
||||
|
||||
The problem is architectural: there is no incremental secondary index
|
||||
keyed by the sort property, only the primary `BTreeMap` keyed by
|
||||
`IndexKey::IngestedLog { sequence, timestamp }` (`src/index_key.rs`,
|
||||
inserted at `engine.rs:156-181`). Every sorted query re-derives params
|
||||
for and re-sorts the *entire* filtered range from scratch, and range
|
||||
selection (e.g. "give me the next page in this order") has no way to
|
||||
seek directly into a sorted position — it can only sort everything then
|
||||
slice.
|
||||
|
||||
## Scope
|
||||
|
||||
- Introduce a "helper index": a secondary `BTreeMap` keyed by
|
||||
`(sort-param-value, ottyl_seq)` (or equivalent composite key so
|
||||
`ottyl_seq` is the tie-break/fallback ordering baked into the key
|
||||
itself, not a post-hoc partition step) per distinct param name that's
|
||||
been sorted by.
|
||||
- Populate a helper index incrementally as lines are ingested (near
|
||||
`engine.rs:156-181`, alongside the primary index insert), for any
|
||||
param name that currently has a live helper index, rather than
|
||||
rebuilding from scratch per query.
|
||||
- Lines missing the sort param must still be captured in the helper
|
||||
index's ordering, falling back to `ottyl_seq` ordering — matching
|
||||
current `sort_missing_fallback` semantics but expressed as part of the
|
||||
index's key/ordering rather than a separate partition-and-append step.
|
||||
- Adapt the `EngineQuery::Logs` sorted branch (`engine.rs:1459-1483`) to
|
||||
do a range walk over the relevant helper index instead of
|
||||
materializing and sorting the full filtered set every time.
|
||||
- Out of scope: the eviction/purge policy for helper indexes that
|
||||
haven't been used recently — that's ticket 061.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Querying the timeline sorted by a declared param returns results
|
||||
ordered correctly, including lines missing that param falling back to
|
||||
`ottyl_seq` order, matching current behavior.
|
||||
- A sorted query no longer re-sorts the full filtered range from scratch
|
||||
on every call once a helper index for that sort key exists — verified
|
||||
by a test/benchmark showing incremental maintenance rather than
|
||||
full materialize-then-sort.
|
||||
- Existing sort-related tests (from ticket 045) continue to pass
|
||||
unmodified in their assertions.
|
||||
|
||||
## Depends on
|
||||
|
||||
None.
|
||||
@@ -1,43 +0,0 @@
|
||||
# 061 — Sort helper indexes need to be purged after a period of no access
|
||||
|
||||
## Summary
|
||||
|
||||
Ticket 060 introduces per-sort-key helper `BTreeMap` indexes. Left
|
||||
unbounded, a user (or dashboard) that sorts by many distinct params over
|
||||
a session accumulates a helper index per param forever, each duplicating
|
||||
the full timeline's worth of keys. There's no existing time-since-access
|
||||
eviction pattern in `engine.rs` to model this on: the only eviction today
|
||||
is `evict_excess_logs` (`src/engine.rs:1356-1363`, called from ingest at
|
||||
`:181` and from `SetMaxLogs` at `:212`), which is a hard count-cap
|
||||
("drop oldest once over `max_logs`"), not an LRU/last-accessed scheme.
|
||||
|
||||
## Scope
|
||||
|
||||
- Track a last-accessed timestamp per helper index (e.g.
|
||||
`HashMap<String, (BTreeMap<...>, Instant)>` or equivalent), updated
|
||||
whenever a query actually reads from that index.
|
||||
- Add a sweep (triggered periodically, or opportunistically on
|
||||
ingest/query — match whichever fits the existing engine actor's
|
||||
message-loop structure) that drops any helper index whose
|
||||
last-accessed time exceeds a fixed duration.
|
||||
- A dropped helper index must be transparently rebuildable: the next
|
||||
query sorted by that param should lazily rebuild the index (a one-time
|
||||
full scan) exactly as if it had never existed, rather than erroring or
|
||||
falling back permanently to unsorted behavior.
|
||||
- Out of scope: making the purge duration user-configurable via
|
||||
`Settings` — pick a reasonable fixed constant unless the ticket
|
||||
implementer finds a strong reason to expose it.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A helper index untouched for longer than the purge duration is
|
||||
removed from engine memory.
|
||||
- Querying a sort key whose helper index was purged transparently
|
||||
rebuilds it and returns correct results, with no user-visible error.
|
||||
- Actively-used helper indexes (queried more recently than the purge
|
||||
duration) are never evicted.
|
||||
|
||||
## Depends on
|
||||
|
||||
Ticket 060 (timeline custom sort needs helper BTreeMap index) — this
|
||||
ticket assumes helper indexes already exist.
|
||||
Reference in New Issue
Block a user