73 lines
3.7 KiB
Markdown
73 lines
3.7 KiB
Markdown
# 077 — Log queries scan the full index instead of seeking by key range
|
|
|
|
## Summary
|
|
|
|
Every log read — including polling reads that only want logs newer
|
|
than `since_sequence` — walks the *entire* retained log range and
|
|
filters with a closure, rather than seeking the `BTreeMap` directly to
|
|
the relevant key range. This is the read-path optimization the
|
|
`left-right`/sort-helper-index work never got around to: iterators
|
|
should narrow the underlying key range as far as the caller's bounds
|
|
allow, and accept range bounds as a parameter instead of scanning from
|
|
`Start`/`MetricBucketSplit` and filtering after the fact.
|
|
|
|
## Where
|
|
|
|
- `EngineSnapshot::logs_range` (`src/engine/snapshot/mod.rs:159-168`)
|
|
always ranges `IndexKey::Start..IndexKey::LogParamSplit` — i.e. every
|
|
ingested log ever retained — with no way for a caller to pass a
|
|
narrower bound.
|
|
- `EngineSnapshot::logs_since` (`src/engine/snapshot/mod.rs:170-181`)
|
|
calls `logs_range()` and then filters by `sequence < before_sequence`
|
|
and a timestamp window in a closure — it has all the information
|
|
needed to seek instead (`before_sequence` bounds `sequence`
|
|
directly), but throws that away and scans linearly regardless of how
|
|
small the actual window is.
|
|
- The unsorted log-query path in `handle_query_inner`
|
|
(`src/engine/snapshot/query.rs:59-65`) does the same thing:
|
|
`snapshot.logs_range().filter(|(sequence, timestamp, _)| { since ...
|
|
start_ms ... end_ms ... })`. On every poll (frontend passes
|
|
`since_sequence` — see `webapp/src/lib/api.ts:263,343`), this walks
|
|
every previously-seen log line again just to throw it away in the
|
|
filter, before even getting to the new ones.
|
|
- `IndexKey::IngestedLog { sequence, timestamp }`
|
|
(`src/engine/index_key.rs:10-13`) derives `Ord` field-order-first on
|
|
`sequence`, so a lower bound of
|
|
`IndexKey::IngestedLog { sequence: since + 1, timestamp: Timestamp::MIN }`
|
|
(or equivalent) is a valid, correct seek — sequence is assigned
|
|
monotonically at ingest (`src/engine/handle.rs:118-119`), so this
|
|
isn't a heuristic, it's exact.
|
|
|
|
## Scope
|
|
|
|
- Change `logs_range` to accept a key range (e.g.
|
|
`RangeBounds<IndexKey>` or a `(Bound<u64>, Bound<u64>)` over
|
|
sequence) instead of hardcoding `Start..LogParamSplit`, so callers
|
|
can seek.
|
|
- Rework `logs_since` and the `EngineQuery::Logs` handler in
|
|
`query.rs:59-65` to build a lower bound from `since_sequence` (and,
|
|
where possible, from `start_ms`/note below) and pass it through,
|
|
instead of filtering post-hoc from the very start of the index.
|
|
- Note: because `IngestedLog` keys sort by `(sequence, timestamp)` and
|
|
not `(timestamp, sequence)`, a `start_ms`/`end_ms`-only query (no
|
|
`since_sequence`) can't be turned into an exact key-range seek from
|
|
this index alone even though ingestion is roughly time-ordered by
|
|
sequence — call this out explicitly rather than seeking incorrectly;
|
|
it may be acceptable to leave that case as a scan, or to binary-search
|
|
for the first sequence at/after `start_ms` since timestamps are
|
|
non-decreasing with sequence in practice.
|
|
- The sorted branch (`query.rs:82-128`) iterates `sort_helper_index`
|
|
directly rather than `logs_range`, so it's less affected, but still
|
|
does a per-candidate `index.get()` (`query.rs:106-107`) with no early
|
|
bound — check whether the helper index itself can be range-bounded
|
|
once `since_sequence`/`start_ms` are known.
|
|
- Add/extend a test asserting that a `since_sequence` poll on a large
|
|
log store does not touch (or does not compare against) log entries
|
|
at or before `since_sequence` — a range-based seek should make this
|
|
observably true, not just faster.
|
|
|
|
## Depends on
|
|
|
|
None. Related to ticket 060/061 (sort helper indexes) — same
|
|
underlying principle (avoid full scans by indexing/seeking).
|