ottyl/ideas/0004-profile-switch-reindexing-cost.md

Idea: profile switching triggers a full synchronous re-index per metric

Original note

"Metrics and dashboards are tied with the profile. However, they are indexed on the engine. Switching profiles means re-indexing." — a description of a real coupling; unclear whether it's being flagged as a performance problem to fix or just an architectural note to be aware of.

Current state (confirmed in code)

• Profile (src/config/profile.rs:309-318) is pure config — params, metrics, dashboards — with no live engine state of its own. Settings::set_active_profile (src/config/profile.rs:489-502) only flips which profile is active and persists to disk; it doesn't touch the engine.
• The actual switch side effect lives in the API layer: activate_profile (src/api/mod.rs:746-768) calls reload_engine_declarations (src/api/mod.rs:712-744), which:
  1. Removes every current param/metric declaration from the engine (EngineMessage::RemoveParamDeclaration/RemoveMetricDeclaration, skipping auto-derived params).
  2. Calls profile.load_into_engine(engine) (src/config/profile.rs:332-384), re-declaring every param/metric in the newly-activated profile.
• There is no dedicated EngineMessage for "switch profile" — it's entirely remove-everything-then-redeclare-everything via the existing declare/remove messages (src/engine/message.rs:30-63 has no such variant).
• Each metric redeclare triggers a full rebuild: declare_metric (src/engine/snapshot/mod.rs:330-335) → resolve_backfill_and_store_metric → backfill_metric (:481-509), which replays all retained log history (self.logs_range()) through apply_metric_to_log for that metric. Raw ingested lines themselves are never cleared on switch — only metric/param declarations and their derived buckets are dropped and rebuilt.
• This all happens synchronously, one metric at a time, inside the engine's single-writer actor loop (src/engine/handle.rs:303-338) — so switching profiles blocks that loop for a duration that scales with retained log volume × number of metrics in the newly-activated profile.

Why this is an idea, not a ticket

Whether this is worth fixing depends on facts not available from reading code alone: typical retained log volume in real usage, typical metric count per profile, and how often users actually switch profiles mid-session. Without knowing the current switch latency is actually noticeable, a fix would be speculative optimization.

Options if this turns out to matter

• Diff-based reload: only remove/re-add metrics whose definition actually changed between the old and new profile, instead of removing everything and redeclaring everything (a metric present unchanged in both profiles currently still gets a full remove + backfill for no reason).
• Make the removal/backfill sequence async relative to the HTTP response, with the UI showing a "reindexing…" state rather than blocking the request (and the engine loop) until complete.
• Lazily backfill only the time range currently being viewed, deferring the rest — a bigger change, and only worth it if the switch cost is dominated by backfill work the user doesn't immediately look at anyway.

Suggested next step

Before scoping a ticket: confirm with a profile that has a realistic number of metrics and log volume whether switch latency is actually perceptible. If it is, the diff-based reload (first option) is the cheapest fix and a reasonable first ticket.

Implications and things to consider

Performance

• Cost today is O(retained lines × metrics in new profile), done synchronously in the single-writer engine actor (src/engine/handle.rs:303-338) — every other engine message (ingest, queries, alert evaluation) queues up behind a profile switch until it finishes, so the blast radius isn't just "the switch feels slow", it's "the whole app appears frozen" for that duration, including live ingestion of new lines arriving mid-switch.
• The diff-based reload option only helps when profiles share metrics — if switching to a profile with a mostly-disjoint metric set (the common case for genuinely different apps/profiles, which is the whole point of profiles), it doesn't reduce the backfill cost much. It's a bigger win for "tweak one metric in the currently active profile and reactivate" workflows than for switching between unrelated profiles.
• If 0001-actor-split happens, this problem's shape changes significantly — a per-profile worker rebuilding its own perspective from a shared primary index could do that rebuild without blocking ingestion or query-serving on the primary at all, which addresses the "whole app freezes" part of this problem as a side effect, even if the total backfill work stays the same. Worth reading that idea alongside this one.

Code structure

• The remove-then-readd flow lives entirely in the API layer (src/api/mod.rs:712-768), not the engine — a diff-based reload would need reload_engine_declarations to compute a diff (metrics present in old profile but not new → remove; new but not old → declare; present in both — compare full declaration equality, not just name, to decide if a changed regex/threshold needs a rebuild) before sending any engine messages. That diff logic is new code, not a variant of existing code.
• Async/non-blocking reload would need a new EngineMessage variant (today there is none for "switch profile" — it's just existing declare/remove messages) if the goal is to let the engine acknowledge the switch request immediately and do backfill work in the background while still serving other queries — this is a bigger structural change than diff-based reload, since the engine's single-actor model currently has no notion of "background work interleaved with message handling."

Functionality

• If reload becomes async/background, the UI needs a defined "reindexing in progress" state — what does a dashboard panel show for a metric that's declared but not yet backfilled? An empty chart is misleading (looks like "no data") versus a genuinely no-data metric.
• Partial-failure semantics: if backfill for metric 3 of 5 fails or is interrupted (e.g. app restart mid-switch), what state is left behind? Today's synchronous approach has no partial-state problem because it either completes or the whole process is still running; async reload introduces a new failure mode that doesn't exist today.

Not yet considered

• No data gathered yet on real profile sizes/log volumes in actual usage (flagged already in "Why this is an idea") — this entire idea is speculative until that's known.
• No consideration of whether profile switching happens automatically (e.g. auto-detected based on the piped command) or is always a manual user action — if it's always manual and rare, the UX cost of a blocking-but-rare operation may be perfectly acceptable and not worth any of the above complexity.