# Idea: additional indexes to speed up reads

## Original note

"Ask AI to come up with additional indexes that will help speed up
reads." — an open-ended research request rather than a scoped
change; this note captures what's already known so a future deep-dive
doesn't start from zero.

## Current index surface (confirmed in code)

- `IndexKey` (`src/engine/index_key.rs`) — the engine's indexing key
  space includes at least `IngestedLog { sequence, timestamp }`,
  `Bucket` (metric bucket data), and `LastMetricEvent`. Ordering is
  primary-`sequence` for logs (see
  [[0003-timeline-ordering-invariant]] for detail).
- `SortHelperKey` (`src/engine/snapshot/sort.rs`) — a secondary index
  keyed by a declared param's value (plus sequence as tie-break),
  built to support `sort_by` queries; has explicit staleness handling
  (`purge_stale_sort_helper_indexes`, referenced from ticket 081's
  history) tied to new bucket ingestion.
- `.claude/CLAUDE.md`'s repowise health summary already flags "19 open
  static I/O-in-loop / N+1 findings" and calls out `src/engine.rs`'s
  `handle_query` as a function/brain-method hotspot — likely overlaps
  with whatever a fresh performance pass on read paths would surface,
  worth checking those findings first (`get_health()` /
  `get_risk(['src/engine.rs'])` via the repowise MCP tools, once the
  repo's index is initialized — it wasn't available this session, see
  note below) rather than starting from a blank slate.
- Related, already-ticketed read-path work referenced in prior ticket
  history: ticket 077/078 (metric bucket range narrowing — "narrow the
  range, don't rescan everything") and ticket 081 (PromQL dashboard
  queries rebuild full series from scratch per poll) — both were
  framed around the same "avoid full rescans" principle this idea is
  chasing generally.

## Why this is an idea, not a ticket

This is explicitly a "come up with ideas" ask, not a pre-scoped
change — it needs a dedicated investigation (likely with actual
profiling under a representative workload, per this project's
`samply`-based profiling convention) rather than guesses from reading
code structure alone. Ticket 081's history is a useful precedent here:
079/080 (also performance tickets) were removed for lacking "profiling
evidence to justify the fix" — so a fix in this space should follow
`samply record` evidence, not speculation.

## Note

The repowise MCP tools reported no index for this repo this session
("This repository has no repowise index yet") despite
`.claude/CLAUDE.md` describing one as current — worth running `repowise
init --yes` (or checking why the existing index isn't being picked up)
before the next performance-focused session, since `get_risk`/
`get_health`/`get_change_risk` would materially speed up this kind of
investigation.

## Suggested next step

Profile a representative workload with `samply` against the read paths
(`Logs`, `Timeline`, dashboard PromQL panel queries) to find actual
bottlenecks, then propose specific new index structures (or narrowings
of existing ones) grounded in what the flamegraph shows — rather than
guessing at indexes with no evidence they'd matter.

## Implications and things to consider

### Performance

- Any new index is a write-time cost paid on every ingested line in
  exchange for a read-time win — worth stating explicitly per proposed
  index, since the engine's ingest path is already doing param
  extraction + metric bucketing + alert evaluation per line
  (see [[0001-actor-split]]'s findings on how much work already happens
  synchronously per line); an expensive new index (e.g. a full-text
  inverted index for regex/text search) could turn ingest into the new
  bottleneck instead of read.
- A text/regex index in particular is a different kind of structure
  than the existing `IndexKey`/`SortHelperKey` model — those are
  ordered key→value indexes; full-text search typically wants an
  inverted index (token → line positions) or a trigram index, which is
  a new subsystem, not a new `IndexKey` variant. Worth scoping as its
  own investigation rather than bundling into "additional indexes"
  generally.
- Memory cost matters as much as CPU cost here — this is a
  for-the-life-of-the-process in-memory tool (per `CLAUDE.md`'s
  description), so any new index's memory footprint competes directly
  with how much log history a user can retain before running out of
  RAM. A `samply` CPU profile alone won't surface this — a memory
  profile of retained-line-count vs. RSS would be needed too.

### Code structure

- New indexes need to hook into the same lifecycle the existing ones
  do: built incrementally at ingest, invalidated/pruned when retention
  drops old lines, and rebuilt on backfill (declare/profile-switch)
  the same way `SortHelperKey`'s `purge_stale_sort_helper_indexes`
  does. A proposal that only covers "how to build it" without covering
  eviction/backfill parity with existing indexes is incomplete.
- If [[0001-actor-split]] happens first, new indexes would likely be
  scoped to whichever worker owns the data they index (e.g. a
  profile-specific index lives with that profile's worker, not the
  primary log worker) — worth sequencing this investigation after that
  architectural decision rather than before, so new indexes aren't
  designed against a data-ownership model that's about to change.

### Functionality

- Read-path speedups are invisible to users unless something is
  currently slow enough to notice — without a concrete complaint or
  profiling evidence of an actual slow query, this risks becoming
  effort spent on indexes nobody was blocked on. The `samply`-first
  approach in "Suggested next step" is meant to guard against exactly
  this.

### Not yet considered

- No investigation yet into what "reads" actually means for the
  intended scope — dashboard panel polling (already covered by ticket
  081's history), the log timeline's default view, ad-hoc filter-bar
  queries, and the MCP server's query surface (`src/api/mcp_server.rs`)
  are all "reads" with potentially different bottlenecks; a single
  "additional indexes" effort may need to be split once profiling
  shows which of these is actually slow.
- No check on whether retention limits already bound the problem —
  if there's a hard cap on retained log lines, some read-path costs may
  already be bounded in practice regardless of indexing, which changes
  how much this is worth investing in.