# Idea: split the engine's single actor into a log-ingestion worker + per-profile worker(s)

## Original note

> This needs to be fleshed out with ai.
>
> The general idea is that the primary engine ingesting worker is
> doing too much. It should only parse logs to extract params.
>
> Since profiles carry additional (non-derived) params, metrics,
> dashboards, and panels - those should be stored separately and can
> be handled by a different worker. The worker would have read access
> to the primary log sequence and derived params index, but would
> construct it's own profile perspective. It has write access and
> then
>
> The profiles need to be informed of new messages to index.
>
> What if message needs to be different format? Thats per-profile.
>
> System param suppressions?
>
> Regex indexing? We probably want text indexing on the primary while
> also
>
> Values in the primary index should be arcs, to prevent cloning
> allocations.

## Resolved decisions (2026-09-07)

Clarified interactively; supersedes the open questions raised below where
they conflict.

- **No profiling gate.** Proceeding on the code-reading evidence already in
  this doc (double/triple string clones per line, full-`EngineSnapshot`
  deep-clone on every `left_right` reconciliation) rather than waiting on a
  `samply` profile first.
- **Three actors, not two.** The split is: **reader actor** (tails
  stdin/file, splits into lines, timestamps, wraps each line in an `Arc`,
  sends downstream) → **primary actor** (ingests timestamped `Arc` lines,
  extracts declared + auto params only, maintains the `param=value` index —
  no metrics, no profile logic) → **profile actor** (holds its own
  `left_right` reader over the primary's index; builds metrics, custom
  params, timeseries, and PromQL evaluation for the one active profile).
  This resolves the "per-profile message format" question below: there is
  no per-profile parsing fork on the primary — format/shape parsing of an
  already-classified line is entirely the profile actor's concern, on top
  of the shared raw text.
- **Still exactly one active profile at a time.** No concurrent
  multi-profile indexing; the split changes *how* the one active profile's
  data is computed relative to the primary, not the user-facing
  single-active-profile model.
- **Alerts move to the profile actor.** Alert rules are profile-scoped and
  read metric buckets, so they move with metric computation rather than
  staying on the primary.
- **Primary→profile signaling:** the primary actor notifies subscribers
  (e.g. a `tokio::sync::watch` carrying the latest processed
  position/index) right after each `left_right` sync completes. The
  profile actor wakes on that signal and re-reads whatever's newly visible
  through its own `left_right` reader — no busy-polling, no second copy of
  the data in transit.
- **Eviction wins over profile-actor lag.** If the profile actor is
  behind (e.g. mid-backfill on a newly declared metric) when the primary
  evicts old lines (`SetMaxLogs`), the primary evicts on schedule
  regardless — the profile actor silently misses data for the evicted
  range. Simplest option, matches today's single-actor retention model.
- **System param suppressions: dropped from scope.** Not needed right now;
  revisit as a separate idea if it becomes a real requirement.
- **Migration path: incremental**, in this order:
  1. Arc-ify stored values (`Arc<str>`/`Arc<String>` for log text and
     metric-captured strings) — independent win, and the load-bearing
     precondition for cheap cross-actor reads in steps 2–3.
  2. Extract metric computation into a separately-callable stage, still
     within the single actor.
  3. Split into the reader/primary/profile actors described above.

## Current architecture (confirmed in code)

- **One process, one actor, today.** `spawn_engine`
  (`src/cli/mod.rs:95`) calls `engine::spawn(...)` exactly once per
  server run. `load_active_profile` (`src/cli/mod.rs:105-124`) loads
  *one* active profile's param/metric declarations into that single
  engine at startup, before ingest starts, with an explicit comment
  explaining why: "there's no way to reprocess earlier lines later, so
  this must happen before `spawn_ingest`" (`src/cli/mod.rs:104-106`).
  All other `engine::spawn` call sites (`src/api/mod.rs:853,1459`,
  several in `src/config/profile.rs`) are test harnesses, not a second
  production engine instance.
- The actor loop is `run()`, `src/engine/handle.rs:130-365` — a single
  `while let Some(message) = rx.recv().await` over one
  `mpsc::Receiver<EngineMessage>` (`:141`). Every mutation (`Ingest`,
  `IngestMetricSample`, `DeclareParam`/`RemoveParamDeclaration`,
  `ClearParams`, `DeclareMetric`/`UpdateMetric`/
  `RemoveMetricDeclaration`, `SetMaxLogs`, `EnsureSortHelperIndex`,
  `Barrier` — full list `src/engine/message.rs:30-63`) funnels through
  this one channel and one task.
- **Queries already bypass this actor.** `EngineQuery`
  (`src/engine/message.rs:12-27`) doesn't go through the mpsc channel
  — `EngineHandle::ask` reads directly off a `left_right`
  `IndexReader`/`ReadHandle` (`src/engine/handle.rs:49-73`, entry point
  `src/api/mod.rs:218-219`, evaluated by `handle_query_ro`,
  `src/engine/snapshot/mod.rs:856-858`). So read-side scaling is
  already decoupled from the actor loop today — this idea is entirely
  about the **write/mutation** path.
- Per-line ingest (`EngineMessage::Ingest` arm,
  `src/engine/handle.rs:143-219`) does, synchronously, in one
  `left_right` write op, per line: ingest-lag/watermark bookkeeping,
  log-stream header detection, storing the line as
  `DataValue::LogLiteral`, auto-param detection
  (`auto_declare_params`), running **every declared metric's regex**
  against the line (`apply_all_metrics` →
  `apply_metric_to_log`/`resolve_param_value`,
  `src/engine/snapshot/mod.rs:409-460,821-851`), sort-helper-index
  maintenance, log eviction, and then — after publish —
  `alerts::evaluate_alerts` against the resulting snapshot
  (`handle.rs:199-219`), which reads metric buckets that were just
  computed in the same message.
- Module size: `src/engine/` is ~5.5k lines excluding tests (`snapshot/
  mod.rs` alone is 1100 lines — the whole `EngineSnapshot` impl per its
  own header comment).
- **No `Arc`, everywhere owned data.** The index is one flat
  `BTreeMap<IndexKey, DataValue>` (`snapshot/mod.rs:43`) of owned
  `String`s (`DataValue::LogLiteral { text: String }`,
  `src/engine/data_value.rs:81`). Per ingested line, the raw text is
  cloned at least twice for storage alone (`handle.rs:175,184-186`),
  and every metric that matches the line clones its own copy of the
  label/value strings it captures (`apply_metric_to_log`,
  `mod.rs:426,439,444`) — with N metrics referencing overlapping
  params, one line produces N independent owned-String allocations of
  essentially the same substrings.
- **`left_right`'s reconciliation deep-clones everything.** After a
  batch of writes, `left_right::Absorb::sync_with` clones the *entire*
  `EngineSnapshot` (`*self = first.clone()`, `handle.rs:29-31`) to
  reconcile the read-side copy — i.e. the "values should be Arc" note
  isn't just about avoiding per-line clones, it's also about avoiding
  a full-`BTreeMap`-of-Strings deep clone on every reconciliation.
- **No text/regex index exists.** `CompiledFilter::Regex(re) =>
  re.is_match(text)` (`snapshot/filter.rs:159`) runs per line at query
  time; `IndexKey` (`src/engine/index_key.rs:8-32`) has no
  inverted/full-text structure. Every filter-bar query re-scans and
  re-matches the whole retained range.
- **Backfill is a full synchronous history replay today, in the same
  actor.** `declare_metric`/`update_metric`
  (`snapshot/mod.rs:330-391`) → `backfill_metric` (`:481-509`) collects
  **all** retained lines into a `Vec` and replays
  `apply_metric_to_log` over every one, inside the single actor's
  message handling — blocking ingest/alerts/other declarations for the
  duration. `load_into_engine` (`src/config/profile.rs:332-384`) does
  this once per metric, awaiting each response, so loading a profile
  with N metrics against L retained lines costs O(N·L) synchronous
  scan time. This is also the mechanism [[0004-profile-switch-reindexing-cost]]
  describes for *runtime* profile switching via the `activate_profile`
  API handler (`src/api/mod.rs:712-768`, which removes then
  re-declares via the same messages) — worth reconciling that this
  research pass separately characterized profile activation at
  startup as one-shot ("no way to reprocess earlier lines later"),
  while the earlier research for ticket 099/idea 0004 confirmed a
  *runtime* reload path exists via `activate_profile`. Both are true:
  startup load is one-shot by necessity (ingest hasn't started yet),
  and the runtime reload path reuses the same declare/backfill
  machinery after the fact — but this is worth double-checking before
  design work starts, since the two research passes framed it
  differently.
- **Alerts are coupled to the single actor's local state.** Alert
  evaluation runs directly in the ingest arm against the just-mutated
  snapshot (`handle.rs:199-211`, `alerts::evaluate_alerts`,
  `src/engine/alerts.rs:211`), reading `alert_states:
  HashMap<String, AlertRuntimeState>` that lives on the single actor
  task's own stack frame (`handle.rs:137`) — there's no mechanism today
  for one actor to read another actor's in-memory state.
- **Existing multi-task precedent is producer-side only.**
  `src/engine/ingest.rs`'s stdin/file-tailing and Prometheus-polling
  tasks each hold a cloned `EngineHandle` and push into the *same*
  actor's mpsc channel (spawned at `src/cli/mod.rs:136-145`) — a
  plausible pattern for "many producers, one consumer," but there's no
  existing precedent for multiple *consuming* actors each owning their
  own `left_right` write side that need to stay coherent with each
  other or with a shared upstream.

## Implications and things to consider

### Performance

- The stated goal — "the primary worker should only parse logs to
  extract params" — would remove per-line metric-regex evaluation
  (`apply_all_metrics`) from the primary's critical path, which is
  real work today (N metrics × regex match per line). That's a
  genuine win *if* profile workers can consume the primary's output
  without re-paying an equivalent cost themselves — but since a
  profile worker still has to run its own metrics' regexes against
  each line to build its own buckets, the total CPU work done across
  the process doesn't shrink; what changes is that it's no longer all
  serialized on one task, so it can run concurrently with the primary
  (a real win on a multi-core box) and a slow/backfilling profile
  worker no longer blocks raw ingestion.
- The "values in the primary index should be Arcs" change is
  independently valuable regardless of whether the actor split
  happens — it directly addresses the confirmed double/triple string
  cloning per line (storage clone + per-metric label/value clones) and
  the full-snapshot deep-clone on every `left_right` reconciliation.
  Worth sequencing as a prerequisite: switching to `Arc<str>` (or
  `Arc<String>`) for stored text means a profile worker reading "the
  primary log sequence" can cheaply hold references to the same
  underlying bytes instead of needing its own copies — this is likely
  a load-bearing precondition for the split being worth it at all, not
  a nice-to-have alongside it.
- A profile worker needing "read access to the primary log sequence
  and derived params index" implies either (a) a second `left_right`
  reader over the primary's structures (cheap, if values are Arc'd —
  see above) or (b) a message-replay feed (the primary notifies
  profile workers of new lines, per the note's "profiles need to be
  informed of new messages to index"). Option (b) reintroduces a
  second data copy per profile worker unless the message itself only
  carries an `Arc`/index reference rather than owned data — otherwise
  this recreates the same N-copies-per-line problem this idea is
  trying to remove, just distributed across processes/tasks instead of
  within one.
- Backfill cost (O(N metrics × L retained lines), see [[0004-profile-switch-reindexing-cost]])
  doesn't disappear with a split — it moves to whichever worker owns
  the profile. The upside is it no longer blocks the *primary's*
  ingest/query-serving while it happens, which is the main perf
  argument for doing this at all, per that idea's "Performance"
  section.
- Full-text/regex indexing on the primary (per the note's "Regex
  indexing?") is a materially different performance investment from
  the Arc/split work — an inverted or trigram index trades ingest-time
  CPU and memory for query-time speed on filter-bar searches, and is
  worth scoping as its own effort (see [[0005-additional-read-indexes]])
  rather than folded into this split, since it doesn't depend on the
  actor topology.

### Code structure

- This is the single largest structural change on the idea list —
  `src/engine/` (~5.5k lines) currently assumes one writer owning one
  `left_right` pair. Splitting introduces: a defined "primary
  perspective" (raw logs + derived/auto params only, no
  profile-specific metrics) as its own bounded module, a
  "profile perspective" abstraction for the one active profile, a
  notification/subscription mechanism from primary to the profile actor
  (the note's "profiles need to be informed of new messages to index" —
  doesn't exist in any form today; the closest analogue, `ingest.rs`'s
  producer tasks feeding one consumer, runs the opposite direction).
  **Resolved** (see "Resolved decisions"): the primary signals via e.g. a
  `tokio::sync::watch` after each `left_right` sync completes, and the
  profile actor re-reads through its own `left_right` reader rather than
  the primary pushing data. A query-fan-out/merge layer is still needed
  for any request
  that needs both raw logs and profile-derived data in one response
  (today `Logs`/`Timeline` queries read both from a single
  `EngineSnapshot` in one pass, e.g. `query.rs:154`'s
  `log_entry_params` call against `compiled_params` from the same
  snapshot — splitting the data sources means this becomes a real join
  across two snapshots/workers instead of one borrow).
- Alert evaluation (`handle.rs:199-211`) currently depends on running
  immediately after metric bucketing in the same message, against the
  same snapshot. **Resolved** (see "Resolved decisions"): alerts move
  with metric computation into the profile actor, since alert rules are
  profile-scoped and read the metric buckets computed there.
- `EngineHandle`'s current API (`handle.rs:49-73` for reads,
  the mpsc `EngineMessage` send for writes) would need to become two
  APIs, or one handle type parameterized by "primary" vs. "profile" —
  worth deciding whether `api.rs`'s handlers (which currently hold one
  `EngineHandle` in `ApiState`) address a single combined handle that
  internally fans out, or explicitly juggle a primary handle plus an
  active-profile handle.
- Given `CLAUDE.md`'s existing note that `engine.rs` is "the file to
  read first" and already the largest/highest-churn file in the repo
  (13 commits/90d per the repowise summary), this split is also a
  legibility opportunity, not just a performance one — today a reader
  has to understand ingest, metrics, alerts, and backfill as one
  intertwined concern to understand any one of them.

### Functionality

- **Per-profile message format** — **resolved**, see "Resolved decisions"
  above: there's no per-profile parsing fork on the primary. Log-stream
  header detection and auto-param extraction stay a one-time, primary-side,
  profile-agnostic concern (`derive_log_stream_params_from_row`,
  `auto_declare_params`); any further message-shape interpretation the
  profile actor needs happens independently, on top of the shared raw
  `Arc` text, in the profile actor itself.
- **System param suppressions** — **dropped from scope** (see "Resolved
  decisions"). Revisit as a separate idea if it becomes a real need.
- **Concurrency** — **resolved**: still strictly one active profile at a
  time. The split changes *how* that one profile's data is computed
  relative to the primary, not the user-facing model — no multi-profile
  concurrent indexing.

### Not yet considered

- **Failure/restart semantics** — partially resolved: the profile actor
  falling behind the primary during a backfill is expected and accepted
  (see "Resolved decisions" — eviction wins over lag, silent gaps for the
  evicted range). Still open: what happens if the profile actor *crashes*
  outright (not just lags) — does it restart and replay from whatever the
  primary currently retains, or is there a persisted checkpoint? Not yet
  decided; likely fine to punt until the split is actually implemented,
  since "restart and rebuild from currently-retained history" is a
  reasonable default consistent with the accepted-gaps decision above.
- **Retention/eviction coordination** — **resolved**, see "Resolved
  decisions": no low-water-mark. The primary evicts on its own schedule
  (`evict_excess_logs`, `handle.rs:195`, `SetMaxLogs`) regardless of the
  profile actor's progress; a lagging profile actor silently misses data
  for the evicted range rather than blocking primary eviction.
- **Testing story**: `src/engine/tests.rs` (2348 lines) and the
  `test_handle` pattern (see ticket 099) are built entirely around one
  actor with one channel — a split roughly doubles the kinds of
  integration tests needed (primary alone, profile worker alone,
  primary+profile interaction) and the existing test harness would
  need real rework, not just extension. Still open — no resolution yet.
- **Migration path** — **resolved**, see "Resolved decisions": incremental,
  in three steps (Arc-ify stored values → extract metric computation into
  a still-single-actor stage → split into reader/primary/profile actors).
- **Whether this is worth it at all without profiling evidence** —
  **resolved**, see "Resolved decisions": proceeding without a `samply`
  profile, on the code-reading evidence already documented in this file.