ottyl/ideas/0004-profile-switch-reindexing-cost.md

# Idea: profile switching triggers a full synchronous re-index per metric

## Original note

"Metrics and dashboards are tied with the profile. However, they are
indexed on the engine. Switching profiles means re-indexing." — a
description of a real coupling; unclear whether it's being flagged as
a performance problem to fix or just an architectural note to be aware
of.

## Current state (confirmed in code)

- `Profile` (`src/config/profile.rs:309-318`) is pure config — params,
  metrics, dashboards — with no live engine state of its own.
  `Settings::set_active_profile` (`src/config/profile.rs:489-502`)
  only flips which profile is active and persists to disk; it doesn't
  touch the engine.
- The actual switch side effect lives in the API layer:
  `activate_profile` (`src/api/mod.rs:746-768`) calls
  `reload_engine_declarations` (`src/api/mod.rs:712-744`), which:
  1. Removes every current param/metric declaration from the engine
     (`EngineMessage::RemoveParamDeclaration`/`RemoveMetricDeclaration`,
     skipping auto-derived params).
  2. Calls `profile.load_into_engine(engine)`
     (`src/config/profile.rs:332-384`), re-declaring every param/metric
     in the newly-activated profile.
- There is no dedicated `EngineMessage` for "switch profile" — it's
  entirely remove-everything-then-redeclare-everything via the
  existing declare/remove messages
  (`src/engine/message.rs:30-63` has no such variant).
- Each metric redeclare triggers a full rebuild: `declare_metric`
  (`src/engine/snapshot/mod.rs:330-335`) →
  `resolve_backfill_and_store_metric` → `backfill_metric`
  (`:481-509`), which replays **all retained log history**
  (`self.logs_range()`) through `apply_metric_to_log` for that metric.
  Raw ingested lines themselves are never cleared on switch — only
  metric/param declarations and their derived buckets are dropped and
  rebuilt.
- This all happens synchronously, one metric at a time, inside the
  engine's single-writer actor loop
  (`src/engine/handle.rs:303-338`) — so switching profiles blocks that
  loop for a duration that scales with retained log volume × number of
  metrics in the newly-activated profile.

## Why this is an idea, not a ticket

Whether this is worth fixing depends on facts not available from
reading code alone: typical retained log volume in real usage, typical
metric count per profile, and how often users actually switch profiles
mid-session. Without knowing the current switch latency is actually
noticeable, a fix would be speculative optimization.

## Options if this turns out to matter

- Diff-based reload: only remove/re-add metrics whose *definition*
  actually changed between the old and new profile, instead of
  removing everything and redeclaring everything (a metric present
  unchanged in both profiles currently still gets a full remove +
  backfill for no reason).
- Make the removal/backfill sequence async relative to the HTTP
  response, with the UI showing a "reindexing…" state rather than
  blocking the request (and the engine loop) until complete.
- Lazily backfill only the time range currently being viewed, deferring
  the rest — a bigger change, and only worth it if the switch cost is
  dominated by backfill work the user doesn't immediately look at
  anyway.

## Suggested next step

Before scoping a ticket: confirm with a profile that has a realistic
number of metrics and log volume whether switch latency is actually
perceptible. If it is, the diff-based reload (first option) is the
cheapest fix and a reasonable first ticket.

## Implications and things to consider

### Performance

- Cost today is O(retained lines × metrics in new profile), done
  synchronously in the single-writer engine actor
  (`src/engine/handle.rs:303-338`) — every other engine message
  (ingest, queries, alert evaluation) queues up behind a profile switch
  until it finishes, so the blast radius isn't just "the switch feels
  slow", it's "the whole app appears frozen" for that duration,
  including live ingestion of new lines arriving mid-switch.
- The diff-based reload option only helps when profiles share metrics
  — if switching to a profile with a mostly-disjoint metric set (the
  common case for genuinely different apps/profiles, which is the
  whole point of profiles), it doesn't reduce the backfill cost much.
  It's a bigger win for "tweak one metric in the currently active
  profile and reactivate" workflows than for switching between
  unrelated profiles.
- If [[0001-actor-split]] happens, this problem's shape changes
  significantly — a per-profile worker rebuilding its own perspective
  from a shared primary index could do that rebuild without blocking
  ingestion or query-serving on the primary at all, which addresses the
  "whole app freezes" part of this problem as a side effect, even if
  the total backfill work stays the same. Worth reading that idea
  alongside this one.

### Code structure

- The remove-then-readd flow lives entirely in the API layer
  (`src/api/mod.rs:712-768`), not the engine — a diff-based reload
  would need `reload_engine_declarations` to compute a diff (metrics
  present in old profile but not new → remove; new but not old →
  declare; present in both — compare full declaration equality, not
  just name, to decide if a changed regex/threshold needs a rebuild)
  before sending any engine messages. That diff logic is new code, not
  a variant of existing code.
- Async/non-blocking reload would need a new `EngineMessage` variant
  (today there is none for "switch profile" — it's just existing
  declare/remove messages) if the goal is to let the engine
  acknowledge the switch request immediately and do backfill work
  in the background while still serving other queries — this is a
  bigger structural change than diff-based reload, since the engine's
  single-actor model currently has no notion of "background work
  interleaved with message handling."

### Functionality

- If reload becomes async/background, the UI needs a defined
  "reindexing in progress" state — what does a dashboard panel show
  for a metric that's declared but not yet backfilled? An empty chart
  is misleading (looks like "no data") versus a genuinely no-data
  metric.
- Partial-failure semantics: if backfill for metric 3 of 5 fails or is
  interrupted (e.g. app restart mid-switch), what state is left behind?
  Today's synchronous approach has no partial-state problem because it
  either completes or the whole process is still running; async reload
  introduces a new failure mode that doesn't exist today.

### Not yet considered

- No data gathered yet on real profile sizes/log volumes in actual
  usage (flagged already in "Why this is an idea") — this entire idea
  is speculative until that's known.
- No consideration of whether profile switching happens automatically
  (e.g. auto-detected based on the piped command) or is always a
  manual user action — if it's always manual and rare, the UX cost of
  a blocking-but-rare operation may be perfectly acceptable and not
  worth any of the above complexity.