ottyl/tickets/077-log-queries-scan-full-index-instead-of-seeking-by-key-range.md

077 — Log queries scan the full index instead of seeking by key range

Summary

Every log read — including polling reads that only want logs newer than since_sequence — walks the entire retained log range and filters with a closure, rather than seeking the BTreeMap directly to the relevant key range. This is the read-path optimization the left-right/sort-helper-index work never got around to: iterators should narrow the underlying key range as far as the caller's bounds allow, and accept range bounds as a parameter instead of scanning from Start/MetricBucketSplit and filtering after the fact.

Where

• EngineSnapshot::logs_range (src/engine/snapshot/mod.rs:159-168) always ranges IndexKey::Start..IndexKey::LogParamSplit — i.e. every ingested log ever retained — with no way for a caller to pass a narrower bound.
• EngineSnapshot::logs_since (src/engine/snapshot/mod.rs:170-181) calls logs_range() and then filters by sequence < before_sequence and a timestamp window in a closure — it has all the information needed to seek instead (before_sequence bounds sequence directly), but throws that away and scans linearly regardless of how small the actual window is.
• The unsorted log-query path in handle_query_inner (src/engine/snapshot/query.rs:59-65) does the same thing: snapshot.logs_range().filter(|(sequence, timestamp, _)| { since ... start_ms ... end_ms ... }). On every poll (frontend passes since_sequence — see webapp/src/lib/api.ts:263,343), this walks every previously-seen log line again just to throw it away in the filter, before even getting to the new ones.
• IndexKey::IngestedLog { sequence, timestamp } (src/engine/index_key.rs:10-13) derives Ord field-order-first on sequence, so a lower bound of IndexKey::IngestedLog { sequence: since + 1, timestamp: Timestamp::MIN } (or equivalent) is a valid, correct seek — sequence is assigned monotonically at ingest (src/engine/handle.rs:118-119), so this isn't a heuristic, it's exact.

Scope

• Change logs_range to accept a key range (e.g. RangeBounds<IndexKey> or a (Bound<u64>, Bound<u64>) over sequence) instead of hardcoding Start..LogParamSplit, so callers can seek.
• Rework logs_since and the EngineQuery::Logs handler in query.rs:59-65 to build a lower bound from since_sequence (and, where possible, from start_ms/note below) and pass it through, instead of filtering post-hoc from the very start of the index.
• Note: because IngestedLog keys sort by (sequence, timestamp) and not (timestamp, sequence), a start_ms/end_ms-only query (no since_sequence) can't be turned into an exact key-range seek from this index alone even though ingestion is roughly time-ordered by sequence — call this out explicitly rather than seeking incorrectly; it may be acceptable to leave that case as a scan, or to binary-search for the first sequence at/after start_ms since timestamps are non-decreasing with sequence in practice.
• The sorted branch (query.rs:82-128) iterates sort_helper_index directly rather than logs_range, so it's less affected, but still does a per-candidate index.get() (query.rs:106-107) with no early bound — check whether the helper index itself can be range-bounded once since_sequence/start_ms are known.
• Add/extend a test asserting that a since_sequence poll on a large log store does not touch (or does not compare against) log entries at or before since_sequence — a range-based seek should make this observably true, not just faster.

Depends on

None. Related to ticket 060/061 (sort helper indexes) — same underlying principle (avoid full scans by indexing/seeking).