ottyl/tickets/080-json-field-filter-reparses-json-per-line-per-query.md

080 — JSON field filters re-parse every candidate line's JSON on every query

Summary

A filter comparing a JSON field (e.g. a log line shaped as JSON, filtered by a field inside it) re-parses the full JSON body of every candidate log line from scratch on every query that uses such a filter — there's no memoization of the parse, even though the same log line's JSON gets re-parsed identically on every subsequent poll until the line ages out.

Where

• eval_json_comparison (src/engine/snapshot/filter.rs:177-185) calls serde_json::from_str::<serde_json::Value>(text) on the raw log text every time it's invoked, and it's invoked once per candidate line per query (called from the per-line filter evaluation path used in query.rs:59-70/111-116 via CompiledFilter::eval).
• Contrast with log_entry_params (used a few lines below in query.rs:79/121), which at least only runs against lines that survive the filter, not before — the JSON parse here runs as part of filtering itself, so it can't be skipped even for lines that get discarded downstream for other reasons if this filter is combined with others.

Scope

• Lowest-effort option: nothing to cache at the snapshot level today since DataValue::LogLiteral only stores raw text (src/engine/snapshot/mod.rs / data_value.rs) — re-parsing per query is currently unavoidable without changing storage. Confirm this is worth doing before storing pre-parsed JSON: check whether most ingested lines are actually JSON-shaped in typical usage, since adding a parsed-JSON cache to every ingested line penalizes the (likely more common) non-JSON case.
• If justified, consider parsing once at ingest time only for lines that look JSON-shaped (reuse the cheap detection already implied by eval_json_comparison's Ok(...) else return false short-circuit) and storing the parsed serde_json::Value (or just the needed field extractions) alongside LogLiteral, invalidated/dropped the same way the line itself is evicted (evict_excess_logs, src/engine/handle.rs:153).
• Alternative, cheaper fix: memoize the parse per-query-invocation rather than per-line-per-filter-eval, i.e. if a single query applies the JSON filter against a line more than once (it currently doesn't appear to, but double-check CompiledFilter composition — AND/OR combinators) avoid re-parsing within that single pass.

Depends on

None.