ottyl/tickets/079-free-text-regex-filter-has-no-index-full-scan-per-query.md

079 — Free-text regex filters have no index; full scan + regex eval per query

Summary

A free-text search-bar filter (a bare regex, not a param[value]op/ottyl_seq/ottyl_ts comparison) is evaluated line-by-line against every retained log on every query — there's no narrower key range to seek into here since the match isn't tied to a declared param or timestamp, so this is the one case in the read path where "narrow the key range" doesn't directly apply and an actual secondary index would be needed instead.

Where

• CompiledFilter::Regex (compiled in src/engine/snapshot/filter.rs, evaluated via .eval(...) per log line) is checked against the full text of every candidate log line returned by logs_range()/the sort-helper iteration in handle_query_inner (src/engine/snapshot/query.rs:59-70 and 111-116) — i.e. once ticket 077 narrows the sequence/time range, a regex-text filter still forces a full per-line regex match over everything left in that range, and if there's no since_sequence bound at all (e.g. a fresh search over the whole retained log window), this is a full linear scan with a regex eval per line.
• There is no cached/precomputed text index (trigram, token, or otherwise) — every query recompiles nothing (the regex itself is cached fine, see param/metric regex caches in src/engine/snapshot/mod.rs:26-34), but re-evaluates the regex against the full text of every line in range, every time.

Scope

• This is the most invasive of the read-path tickets — flag as such. Do not implement speculatively; first confirm via profiling (samply record against a realistic log volume with an active free-text filter) that this is actually the dominant cost before building an index, since regex eval over a bounded/paginated result set may already be fast enough once ticket 077's range-narrowing lands.
• If profiling confirms it's worth it: investigate a coarse pre-filter (e.g. a trigram index or per-line token set) built incrementally at ingest time (alongside update_sort_helper_indexes_on_ingest, src/engine/handle.rs:151) that can cheaply reject lines that can't possibly match a given regex's required literal substrings, before paying for the actual regex match — not a replacement for the regex eval, a way to skip most of it.
• Keep the index bounded in size / evictable in step with evict_excess_logs (src/engine/handle.rs:153) so it doesn't grow unbounded relative to max logs in memory.

Depends on

Should land after ticket 077 (range narrowing) so profiling reflects the scan cost that's actually left, not the one about to be fixed anyway.