Bound auto-declared param cardinality and recognize log stream format (ticket 050/054)

auto_declare_params (src/engine.rs) previously never overwrote or pruned a
declaration, so a burst of unusual key=value-shaped lines could permanently
inflate the params UI forever. A real macOS `log stream` dump
(huig_dump.txt, copied into src/fixtures/log_stream_sample.txt) discovered
220 distinct auto-declared params from under 700 lines, mostly one-off
WiFi/radio stat counters (20+ pairs on a single line), plus base64-blob
false positives from key()'s greedy alnum matching landing right before
`==` padding.

Fix, combining two bounds per the ticket's options:
- src/logfmt.rs: extract_pairs/pair now rejects a candidate pair whose
  bareword value starts with a bare '=' (looks_like_base64_padding_artifact)
  — the exact shape produced when a base64/hash blob's padding sits right
  after a run of alnum chars that key() consumed as if it were a key.
- src/engine.rs: auto_declare_params now enforces a per-line cap
  (MAX_NEW_AUTO_PARAMS_PER_LINE = 8) on new params from one line, and a
  total ceiling (MAX_AUTO_DECLARED_PARAMS = 64) on auto-derived params
  overall — the per-line cap alone doesn't bound unbounded key *variety*
  across many lines, so both are needed.

Also adds structured extraction for log stream's fixed-column shape
(Timestamp Thread Type Activity PID TTL, then `process: message`):
detect_format gains a new "log_stream" ottyl_fmt value (checked first,
since it's the most specific of the four), and
auto_declare_log_stream_params declares log_type/process/pid as standalone
queryable params the first time such a line is seen — derived positionally
from the column layout rather than =-scanning, so there's no ambiguity or
false-positive risk. This is a separate always-attempted extraction
alongside the existing key=value pass, not a replacement for it: a log
stream line's message tail can still contain real key=value pairs (e.g.
the WiFi stat lines) that auto_declare_params keeps discovering
independently. This also fully addresses ticket 054, confirming huig_dump.txt
is space-padded fixed-width columns, not TSV, and that no separate TSV
format is needed.

Throughput (src/fixtures/log_stream_sample.txt x50, release build):
before (caps disabled) ~369k lines/sec, 206 auto-declared params;
after (caps enabled) ~1.20M lines/sec, 64 auto-declared params — about
3.2x faster, confirming the ever-growing declaration set was the
bottleneck rather than per-line regex/nom scanning cost.

Removes tickets/050 and tickets/054 now that both are implemented.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o

tickets/050-structured-header-extraction-and-bounded-logfmt-overfit.md

@@ -1,84 +0,0 @@
-# 050 — Bound logfmt-overfit param cardinality; derive params from `log stream`-style structured headers
-
-## Summary
-
-A real-world `log stream` (macOS unified log) dump (`huig_dump.txt`, ~700
-lines) exposes two related problems in the auto-param-derivation path
-(`auto_declare_params` in `src/engine.rs`, backed by `extract_pairs` in
-`src/logfmt.rs`, run on *every* ingested line at `src/engine.rs:155`):
-
-1. **`=`-overfitting produces runaway, low-value param cardinality.**
-   `extract_pairs` scans anywhere in the line for `key=value`/`key="value"`
-   and treats every match as a declarable param. On the sample dump this
-   discovers **220 distinct auto-declared param names** from under 700
-   lines — one-off WiFi/radio stat dumps alone contribute lines with
-   20-27 `key=value` pairs each (e.g. `txRTSFrm=169 txRTSFail=0
-   rxCTSUcast=168 ...`), most of which are internal counters that will
-   never recur meaningfully. Some matches are outright false positives:
-   base64-encoded blobs ending in padding (e.g.
-   `rKTEtG7JmnThhoKPbZiVrA=='>,`) get parsed as a `key=value` pair because
-   `=` precedes non-whitespace. Since `auto_declare_params` "never
-   overwrites an existing declaration" and there is no cap or pruning
-   anywhere in the ingest path, every distinct key seen ever becomes a
-   permanent declaration — this is almost certainly the source of the
-   reported ingest lag on this input (per-line nom scanning cost is
-   presumably fine; the unbounded, ever-growing declaration/index set is
-   the more likely culprit and should be measured either way) and it
-   clutters the params UI with noise no user asked for.
-
-2. **The input's real structure is being ignored.** `log stream` output is
-   not logfmt at all — it's a fixed-column header (`Timestamp`, `Thread`,
-   `Type`, `Activity`, `PID`, `TTL`) followed by a free-text
-   `process: message` tail (see header row and body of `huig_dump.txt`).
-   `Type` (`Default`/`Error`/`Activity`), the process name (`kernel`,
-   `airportd`, `bluetoothd`, `WindowServer`, ...), and `PID` are
-   low-cardinality, genuinely useful params that could be derived
-   directly and cheaply from column position — with no `=`-scanning, no
-   ambiguity, and no risk of false positives — instead of (or in addition
-   to) whatever generic `key=value` pairs happen to appear later in the
-   line.
-
-## Scope
-
-- Add a bound/safeguard on `auto_declare_params` so a single burst of
-  unusual lines can't permanently inflate the declared-param set:
-  options include (pick and justify one, or combine) a cap on the total
-  number of auto-derived declarations, a per-line cap on how many
-  `key=value` pairs from one line are eligible for auto-declaration, and/or
-  a heuristic to reject candidate pairs that look like base64 fragments
-  (trailing `=`/`==` with a base64-alphabet value and no realistic bareword
-  shape) in `src/logfmt.rs`'s `extract_pairs`/`pair`.
-- Add an optional structured-header extractor for `log stream`-shaped
-  input: detect the fixed-column header/row shape (timestamp, thread id,
-  log type, activity id, pid, ttl, `process: message`) and surface
-  `Type`/process-name/`PID` as derivable params, analogous to how
-  `ottyl_fmt` (ticket 044) records which format a line matched. Decide
-  whether this is a new `ottyl_fmt` variant (e.g. `"log_stream"`) or a
-  separate always-attempted extraction that runs alongside
-  logfmt/JSON detection, and record the decision in the PR.
-- Use `huig_dump.txt` (already in the repo root) as the seed fixture for
-  tests — copy the relevant lines into a test fixture under wherever
-  existing ingest fixtures live, rather than reading the file from the
-  repo root at test time.
-- Re-measure ingest throughput on this fixture before/after to confirm
-  the lag is actually addressed, and note the before/after numbers in the
-  PR description.
-
-## Acceptance criteria
-
-- Ingesting `huig_dump.txt`'s lines no longer produces hundreds of
-  permanent auto-declared params from one-off stat-dump lines; the bound
-  chosen is verified by a test asserting the declared-param count stays
-  under a stated threshold after ingesting the fixture.
-- Base64-padding false positives (e.g. the `==`-terminated blobs in the
-  fixture) are not auto-declared as params.
-- `Type`, process name, and `PID` are available as derivable/queryable
-  params for `log stream`-shaped lines, verified against the fixture.
-- Existing logfmt/JSON auto-declaration behavior for non-`log
-  stream`-shaped lines (existing tests in `src/logfmt.rs`/`src/engine.rs`)
-  is unchanged.
-
-## Depends on
-
-044 (reserved param naming / `ottyl_fmt` — this ticket likely extends the
-same format-detection mechanism).

tickets/054-log-stream-format-detection.md

@@ -1,63 +0,0 @@
-# 054 — `log stream` output isn't recognized as a known format (folds into ticket 050)
-
-## Summary
-
-The report was "`log stream |& ottyl` should capture the format
-properly... tab separated format is being ignored, add it as a known
-type." Investigating `huig_dump.txt` (a real `log stream` capture already
-in the repo root) shows the premise needs correcting: the file is **not**
-tab-separated. `grep -P "\t"` finds a literal tab character on only 3 of
-697 lines, and none of those are the header or a representative data row
-— they're incidental (two `libinfo check path:` continuation lines, one
-WiFi stat line with an embedded tab mid-message). `od -c` on the header
-and data rows confirms the real structure is **space-padded fixed-width
-columns** (`Timestamp`, `Thread`, `Type`, `Activity`, `PID`, `TTL`,
-followed by a free-text `process: message` tail) — visually aligned
-columns that read as tab-separated but aren't.
-
-There is no TSV handling anywhere in the codebase today: `src/logfmt.rs`,
-`src/engine.rs`, and `src/ingest.rs` contain zero references to tabs or
-TSV. Format detection is entirely `detect_format` (`src/logfmt.rs:70-78`):
-try JSON, else fall back to `extract_pairs` (`logfmt.rs:45-59`) for
-logfmt, else `"none"` — no third branch exists, and `ottyl_fmt`
-(`src/engine.rs:1005`, filter compilation at `engine.rs:775`, `824`) only
-ever reports one of those three values.
-
-This is a duplicate of, not distinct from, ticket 050
-(`tickets/050-structured-header-extraction-and-bounded-logfmt-overfit.md`),
-which already identifies the exact same root cause — `log stream`
-output "is not logfmt at all — it's a fixed-column header... followed by
-a free-text `process: message` tail" — and already scopes a structured-
-header extractor plus the question of whether it becomes a new
-`ottyl_fmt` variant (e.g. `"log_stream"`). The reporter most likely
-visually misread the aligned whitespace columns as tab-separated.
-
-## Scope
-
-- Do not implement a separate "TSV `ottyl_fmt` type" — there is no
-  evidence of genuinely tab-delimited `log stream` output in this
-  codebase's fixtures, and building detection for a format that doesn't
-  occur would be speculative.
-- Close/merge this wishlist item into ticket 050's existing scope: its
-  structured-header extractor for `log stream`-shaped input already
-  covers "capturing the format properly" for this input shape.
-- If the user can supply a genuinely tab-delimited sample (e.g. from a
-  different `log stream --style` flag, or a shell pipeline that
-  reformats the output with real tabs), re-open this as a distinct
-  ticket scoped to that concrete input — file:line references above
-  should make it easy to confirm one way or the other before implementing
-  anything.
-
-## Acceptance criteria
-
-- No code changes result directly from this ticket; instead, ticket 050
-  is updated (or its implementation, when picked up, is confirmed) to
-  cover recognizing `log stream`'s fixed-column shape as a known format.
-- If a true TSV sample is later provided, a new ticket is filed against
-  it specifically, rather than reopening this one with an unconfirmed
-  premise.
-
-## Depends on
-
-050 (this ticket's concern is already inside 050's scope; treat as
-duplicate/fold-in rather than independent work).