Add tickets and ideas

ideas/0004-profile-switch-reindexing-cost.md

@@ -0,0 +1,142 @@
+# Idea: profile switching triggers a full synchronous re-index per metric
+
+## Original note
+
+"Metrics and dashboards are tied with the profile. However, they are
+indexed on the engine. Switching profiles means re-indexing." — a
+description of a real coupling; unclear whether it's being flagged as
+a performance problem to fix or just an architectural note to be aware
+of.
+
+## Current state (confirmed in code)
+
+- `Profile` (`src/config/profile.rs:309-318`) is pure config — params,
+  metrics, dashboards — with no live engine state of its own.
+  `Settings::set_active_profile` (`src/config/profile.rs:489-502`)
+  only flips which profile is active and persists to disk; it doesn't
+  touch the engine.
+- The actual switch side effect lives in the API layer:
+  `activate_profile` (`src/api/mod.rs:746-768`) calls
+  `reload_engine_declarations` (`src/api/mod.rs:712-744`), which:
+  1. Removes every current param/metric declaration from the engine
+     (`EngineMessage::RemoveParamDeclaration`/`RemoveMetricDeclaration`,
+     skipping auto-derived params).
+  2. Calls `profile.load_into_engine(engine)`
+     (`src/config/profile.rs:332-384`), re-declaring every param/metric
+     in the newly-activated profile.
+- There is no dedicated `EngineMessage` for "switch profile" — it's
+  entirely remove-everything-then-redeclare-everything via the
+  existing declare/remove messages
+  (`src/engine/message.rs:30-63` has no such variant).
+- Each metric redeclare triggers a full rebuild: `declare_metric`
+  (`src/engine/snapshot/mod.rs:330-335`) →
+  `resolve_backfill_and_store_metric` → `backfill_metric`
+  (`:481-509`), which replays **all retained log history**
+  (`self.logs_range()`) through `apply_metric_to_log` for that metric.
+  Raw ingested lines themselves are never cleared on switch — only
+  metric/param declarations and their derived buckets are dropped and
+  rebuilt.
+- This all happens synchronously, one metric at a time, inside the
+  engine's single-writer actor loop
+  (`src/engine/handle.rs:303-338`) — so switching profiles blocks that
+  loop for a duration that scales with retained log volume × number of
+  metrics in the newly-activated profile.
+
+## Why this is an idea, not a ticket
+
+Whether this is worth fixing depends on facts not available from
+reading code alone: typical retained log volume in real usage, typical
+metric count per profile, and how often users actually switch profiles
+mid-session. Without knowing the current switch latency is actually
+noticeable, a fix would be speculative optimization.
+
+## Options if this turns out to matter
+
+- Diff-based reload: only remove/re-add metrics whose *definition*
+  actually changed between the old and new profile, instead of
+  removing everything and redeclaring everything (a metric present
+  unchanged in both profiles currently still gets a full remove +
+  backfill for no reason).
+- Make the removal/backfill sequence async relative to the HTTP
+  response, with the UI showing a "reindexing…" state rather than
+  blocking the request (and the engine loop) until complete.
+- Lazily backfill only the time range currently being viewed, deferring
+  the rest — a bigger change, and only worth it if the switch cost is
+  dominated by backfill work the user doesn't immediately look at
+  anyway.
+
+## Suggested next step
+
+Before scoping a ticket: confirm with a profile that has a realistic
+number of metrics and log volume whether switch latency is actually
+perceptible. If it is, the diff-based reload (first option) is the
+cheapest fix and a reasonable first ticket.
+
+## Implications and things to consider
+
+### Performance
+
+- Cost today is O(retained lines × metrics in new profile), done
+  synchronously in the single-writer engine actor
+  (`src/engine/handle.rs:303-338`) — every other engine message
+  (ingest, queries, alert evaluation) queues up behind a profile switch
+  until it finishes, so the blast radius isn't just "the switch feels
+  slow", it's "the whole app appears frozen" for that duration,
+  including live ingestion of new lines arriving mid-switch.
+- The diff-based reload option only helps when profiles share metrics
+  — if switching to a profile with a mostly-disjoint metric set (the
+  common case for genuinely different apps/profiles, which is the
+  whole point of profiles), it doesn't reduce the backfill cost much.
+  It's a bigger win for "tweak one metric in the currently active
+  profile and reactivate" workflows than for switching between
+  unrelated profiles.
+- If [[0001-actor-split]] happens, this problem's shape changes
+  significantly — a per-profile worker rebuilding its own perspective
+  from a shared primary index could do that rebuild without blocking
+  ingestion or query-serving on the primary at all, which addresses the
+  "whole app freezes" part of this problem as a side effect, even if
+  the total backfill work stays the same. Worth reading that idea
+  alongside this one.
+
+### Code structure
+
+- The remove-then-readd flow lives entirely in the API layer
+  (`src/api/mod.rs:712-768`), not the engine — a diff-based reload
+  would need `reload_engine_declarations` to compute a diff (metrics
+  present in old profile but not new → remove; new but not old →
+  declare; present in both — compare full declaration equality, not
+  just name, to decide if a changed regex/threshold needs a rebuild)
+  before sending any engine messages. That diff logic is new code, not
+  a variant of existing code.
+- Async/non-blocking reload would need a new `EngineMessage` variant
+  (today there is none for "switch profile" — it's just existing
+  declare/remove messages) if the goal is to let the engine
+  acknowledge the switch request immediately and do backfill work
+  in the background while still serving other queries — this is a
+  bigger structural change than diff-based reload, since the engine's
+  single-actor model currently has no notion of "background work
+  interleaved with message handling."
+
+### Functionality
+
+- If reload becomes async/background, the UI needs a defined
+  "reindexing in progress" state — what does a dashboard panel show
+  for a metric that's declared but not yet backfilled? An empty chart
+  is misleading (looks like "no data") versus a genuinely no-data
+  metric.
+- Partial-failure semantics: if backfill for metric 3 of 5 fails or is
+  interrupted (e.g. app restart mid-switch), what state is left behind?
+  Today's synchronous approach has no partial-state problem because it
+  either completes or the whole process is still running; async reload
+  introduces a new failure mode that doesn't exist today.
+
+### Not yet considered
+
+- No data gathered yet on real profile sizes/log volumes in actual
+  usage (flagged already in "Why this is an idea") — this entire idea
+  is speculative until that's known.
+- No consideration of whether profile switching happens automatically
+  (e.g. auto-detected based on the piped command) or is always a
+  manual user action — if it's always manual and rare, the UX cost of
+  a blocking-but-rare operation may be perfectly acceptable and not
+  worth any of the above complexity.