Bound auto-declared param cardinality and recognize log stream format (ticket 050/054)

auto_declare_params (src/engine.rs) previously never overwrote or pruned a
declaration, so a burst of unusual key=value-shaped lines could permanently
inflate the params UI forever. A real macOS `log stream` dump
(huig_dump.txt, copied into src/fixtures/log_stream_sample.txt) discovered
220 distinct auto-declared params from under 700 lines, mostly one-off
WiFi/radio stat counters (20+ pairs on a single line), plus base64-blob
false positives from key()'s greedy alnum matching landing right before
`==` padding.

Fix, combining two bounds per the ticket's options:
- src/logfmt.rs: extract_pairs/pair now rejects a candidate pair whose
  bareword value starts with a bare '=' (looks_like_base64_padding_artifact)
  — the exact shape produced when a base64/hash blob's padding sits right
  after a run of alnum chars that key() consumed as if it were a key.
- src/engine.rs: auto_declare_params now enforces a per-line cap
  (MAX_NEW_AUTO_PARAMS_PER_LINE = 8) on new params from one line, and a
  total ceiling (MAX_AUTO_DECLARED_PARAMS = 64) on auto-derived params
  overall — the per-line cap alone doesn't bound unbounded key *variety*
  across many lines, so both are needed.

Also adds structured extraction for log stream's fixed-column shape
(Timestamp Thread Type Activity PID TTL, then `process: message`):
detect_format gains a new "log_stream" ottyl_fmt value (checked first,
since it's the most specific of the four), and
auto_declare_log_stream_params declares log_type/process/pid as standalone
queryable params the first time such a line is seen — derived positionally
from the column layout rather than =-scanning, so there's no ambiguity or
false-positive risk. This is a separate always-attempted extraction
alongside the existing key=value pass, not a replacement for it: a log
stream line's message tail can still contain real key=value pairs (e.g.
the WiFi stat lines) that auto_declare_params keeps discovering
independently. This also fully addresses ticket 054, confirming huig_dump.txt
is space-padded fixed-width columns, not TSV, and that no separate TSV
format is needed.

Throughput (src/fixtures/log_stream_sample.txt x50, release build):
before (caps disabled) ~369k lines/sec, 206 auto-declared params;
after (caps enabled) ~1.20M lines/sec, 64 auto-declared params — about
3.2x faster, confirming the ever-growing declaration set was the
bottleneck rather than per-line regex/nom scanning cost.

Removes tickets/050 and tickets/054 now that both are implemented.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
Hugh Kaznowski
2026-09-06 09:42:50 +01:00
co-authored by Claude Sonnet 5
parent d17cfe9b83
commit f48d3c6ffb
6 changed files with 973 additions and 154 deletions
+4 -1
View File
@@ -104,7 +104,10 @@ const SECTIONS: DocSection[] = [
<p>
<Code>ottyl_seq</Code>, <Code>ottyl_ts</Code>, and <Code>ottyl_fmt</Code> are reserved metric names present
on every log line, so they're always filterable without declaring anything. <Code>ottyl_fmt</Code> holds the
auto-detected format of the line's text — <Code>json</Code>, <Code>logfmt</Code>, or <Code>none</Code>.
auto-detected format of the line's text — <Code>log_stream</Code> (macOS <Code>log stream</Code> output),{' '}
<Code>json</Code>, <Code>logfmt</Code>, or <Code>none</Code>. A <Code>log_stream</Code> line also gets{' '}
<Code>log_type</Code>, <Code>process</Code>, and <Code>pid</Code> params derived automatically from its
fixed-column header.
</p>
<p className="font-semibold text-text">Date/time formats</p>
<p>Timestamp comparisons accept three forms:</p>