ottyl/tickets/059-log-stream-parsing-should-be-generic-whitespace-columnar.md

059 — log stream parsing is one hardcoded regex, not generic whitespace-columnar detection

Summary

Current log stream support is a single hardcoded regex, LOG_STREAM_RE (src/logfmt.rs:93-113), matching exactly six fixed columns (Timestamp Thread Type Activity PID TTL) by content shape (dates/hex/decimals) rather than by parsing the header row generically. auto_declare_log_stream_params (src/engine.rs:451-472) then extracts only three hardcoded fields (log_type, process, pid) via three more bespoke regexes (LOG_STREAM_TYPE_PATTERN etc., logfmt.rs:108-113).

The fixture src/fixtures/log_stream_sample.txt has a real header row (Timestamp Thread Type Activity PID TTL ) whose columns are separated by runs of 2+ spaces — the general CSV-like shape the wishlist asks for — plus a trailing unlabelled process: message tail that must not get merged into the preceding column. Today nothing derives column names or positions from that header; every field is matched by a fixed, hand-tuned regex, so any deviation from this exact log stream output shape silently fails to parse.

Scope

• Replace LOG_STREAM_RE/LOG_STREAM_*_PATTERN's fixed-shape matching with a generic whitespace-columnar detector: given a header line, split on runs of duplicated whitespace (2+ spaces, or a single tab, or " \t " — any whitespace repeated) to determine column names and positions, then apply the same column boundaries to subsequent data lines.
  • The header's last named column's value must still capture through to end-of-line without swallowing the trailing unlabelled message column that log stream appends after the last header-named field (i.e. that trailing text becomes its own unlabelled/message param, not merged into the last header column).
  • Single tabs are acceptable separators even where surrounding spaces aren't doubled.
• Wire the new column-driven extraction into auto_declare_log_stream_params in place of the three bespoke per-field regexes, so declared params are derived from whatever columns the header actually names, not a fixed list of three.
• Use src/fixtures/log_stream_sample.txt as the reference fixture and add/extend tests in src/logfmt.rs/src/engine.rs asserting header columns are correctly detected and the trailing message field is kept separate.
• Out of scope: changing param-cardinality bounding introduced in ticket 050/054 (commit f48d3c6f) — this is purely about how columns are detected/split, not about pruning noisy auto-declarations.

Acceptance criteria

• Piping real log stream output (as in log_stream_sample.txt) through ottyl declares params matching the header's actual column names, not the previous fixed three.
• The trailing unlabelled message text on each log stream line is captured as its own field and never merged into the last named header column.
• A header/data line whose columns are separated by a single tab is parsed identically to one separated by 2+ spaces.

Depends on

None.