58 lines
2.7 KiB
Markdown
58 lines
2.7 KiB
Markdown
# 060 — Timeline sort-by-property does a full materialize+sort per query instead of using a helper index
|
|
|
|
## Summary
|
|
|
|
Sorting the timeline by an arbitrary property does work today (from
|
|
ticket 045): in the `EngineQuery::Logs` branch (`src/engine.rs:1408-1487`),
|
|
when `sort_by` is set, every filtered line is materialized into a
|
|
`Vec<LogEntryDto>` (deriving params for all of them), then sorted
|
|
in-memory with `.sort_by(...)`, with `sort_missing_fallback` (default
|
|
true) partitioning entries that have the sort param from those that
|
|
don't and ordering the latter by `ottyl_seq` (`:1468-1482`).
|
|
|
|
The problem is architectural: there is no incremental secondary index
|
|
keyed by the sort property, only the primary `BTreeMap` keyed by
|
|
`IndexKey::IngestedLog { sequence, timestamp }` (`src/index_key.rs`,
|
|
inserted at `engine.rs:156-181`). Every sorted query re-derives params
|
|
for and re-sorts the *entire* filtered range from scratch, and range
|
|
selection (e.g. "give me the next page in this order") has no way to
|
|
seek directly into a sorted position — it can only sort everything then
|
|
slice.
|
|
|
|
## Scope
|
|
|
|
- Introduce a "helper index": a secondary `BTreeMap` keyed by
|
|
`(sort-param-value, ottyl_seq)` (or equivalent composite key so
|
|
`ottyl_seq` is the tie-break/fallback ordering baked into the key
|
|
itself, not a post-hoc partition step) per distinct param name that's
|
|
been sorted by.
|
|
- Populate a helper index incrementally as lines are ingested (near
|
|
`engine.rs:156-181`, alongside the primary index insert), for any
|
|
param name that currently has a live helper index, rather than
|
|
rebuilding from scratch per query.
|
|
- Lines missing the sort param must still be captured in the helper
|
|
index's ordering, falling back to `ottyl_seq` ordering — matching
|
|
current `sort_missing_fallback` semantics but expressed as part of the
|
|
index's key/ordering rather than a separate partition-and-append step.
|
|
- Adapt the `EngineQuery::Logs` sorted branch (`engine.rs:1459-1483`) to
|
|
do a range walk over the relevant helper index instead of
|
|
materializing and sorting the full filtered set every time.
|
|
- Out of scope: the eviction/purge policy for helper indexes that
|
|
haven't been used recently — that's ticket 061.
|
|
|
|
## Acceptance criteria
|
|
|
|
- Querying the timeline sorted by a declared param returns results
|
|
ordered correctly, including lines missing that param falling back to
|
|
`ottyl_seq` order, matching current behavior.
|
|
- A sorted query no longer re-sorts the full filtered range from scratch
|
|
on every call once a helper index for that sort key exists — verified
|
|
by a test/benchmark showing incremental maintenance rather than
|
|
full materialize-then-sort.
|
|
- Existing sort-related tests (from ticket 045) continue to pass
|
|
unmodified in their assertions.
|
|
|
|
## Depends on
|
|
|
|
None.
|