ottyl/tickets/060-timeline-custom-sort-needs-helper-btreemap-index.md

060 — Timeline sort-by-property does a full materialize+sort per query instead of using a helper index

Summary

Sorting the timeline by an arbitrary property does work today (from ticket 045): in the EngineQuery::Logs branch (src/engine.rs:1408-1487), when sort_by is set, every filtered line is materialized into a Vec<LogEntryDto> (deriving params for all of them), then sorted in-memory with .sort_by(...), with sort_missing_fallback (default true) partitioning entries that have the sort param from those that don't and ordering the latter by ottyl_seq (:1468-1482).

The problem is architectural: there is no incremental secondary index keyed by the sort property, only the primary BTreeMap keyed by IndexKey::IngestedLog { sequence, timestamp } (src/index_key.rs, inserted at engine.rs:156-181). Every sorted query re-derives params for and re-sorts the entire filtered range from scratch, and range selection (e.g. "give me the next page in this order") has no way to seek directly into a sorted position — it can only sort everything then slice.

Scope

• Introduce a "helper index": a secondary BTreeMap keyed by (sort-param-value, ottyl_seq) (or equivalent composite key so ottyl_seq is the tie-break/fallback ordering baked into the key itself, not a post-hoc partition step) per distinct param name that's been sorted by.
• Populate a helper index incrementally as lines are ingested (near engine.rs:156-181, alongside the primary index insert), for any param name that currently has a live helper index, rather than rebuilding from scratch per query.
• Lines missing the sort param must still be captured in the helper index's ordering, falling back to ottyl_seq ordering — matching current sort_missing_fallback semantics but expressed as part of the index's key/ordering rather than a separate partition-and-append step.
• Adapt the EngineQuery::Logs sorted branch (engine.rs:1459-1483) to do a range walk over the relevant helper index instead of materializing and sorting the full filtered set every time.
• Out of scope: the eviction/purge policy for helper indexes that haven't been used recently — that's ticket 061.

Acceptance criteria

• Querying the timeline sorted by a declared param returns results ordered correctly, including lines missing that param falling back to ottyl_seq order, matching current behavior.
• A sorted query no longer re-sorts the full filtered range from scratch on every call once a helper index for that sort key exists — verified by a test/benchmark showing incremental maintenance rather than full materialize-then-sort.
• Existing sort-related tests (from ticket 045) continue to pass unmodified in their assertions.

Depends on

None.