ottyl/ideas/0001-actor-split.md

Idea: split the engine's single actor into a log-ingestion worker + per-profile worker(s)

Original note

This needs to be fleshed out with ai.

The general idea is that the primary engine ingesting worker is doing too much. It should only parse logs to extract params.

Since profiles carry additional (non-derived) params, metrics, dashboards, and panels - those should be stored separately and can be handled by a different worker. The worker would have read access to the primary log sequence and derived params index, but would construct it's own profile perspective. It has write access and then

The profiles need to be informed of new messages to index.

What if message needs to be different format? Thats per-profile.

System param suppressions?

Regex indexing? We probably want text indexing on the primary while also

Values in the primary index should be arcs, to prevent cloning allocations.

Resolved decisions (2026-09-07)

Clarified interactively; supersedes the open questions raised below where they conflict.

• No profiling gate. Proceeding on the code-reading evidence already in this doc (double/triple string clones per line, full-EngineSnapshot deep-clone on every left_right reconciliation) rather than waiting on a samply profile first.
• Three actors, not two. The split is: reader actor (tails stdin/file, splits into lines, timestamps, wraps each line in an Arc, sends downstream) → primary actor (ingests timestamped Arc lines, extracts declared + auto params only, maintains the param=value index — no metrics, no profile logic) → profile actor (holds its own left_right reader over the primary's index; builds metrics, custom params, timeseries, and PromQL evaluation for the one active profile). This resolves the "per-profile message format" question below: there is no per-profile parsing fork on the primary — format/shape parsing of an already-classified line is entirely the profile actor's concern, on top of the shared raw text.
• Still exactly one active profile at a time. No concurrent multi-profile indexing; the split changes how the one active profile's data is computed relative to the primary, not the user-facing single-active-profile model.
• Alerts move to the profile actor. Alert rules are profile-scoped and read metric buckets, so they move with metric computation rather than staying on the primary.
• Primary→profile signaling: the primary actor notifies subscribers (e.g. a tokio::sync::watch carrying the latest processed position/index) right after each left_right sync completes. The profile actor wakes on that signal and re-reads whatever's newly visible through its own left_right reader — no busy-polling, no second copy of the data in transit.
• Eviction wins over profile-actor lag. If the profile actor is behind (e.g. mid-backfill on a newly declared metric) when the primary evicts old lines (SetMaxLogs), the primary evicts on schedule regardless — the profile actor silently misses data for the evicted range. Simplest option, matches today's single-actor retention model.
• System param suppressions: dropped from scope. Not needed right now; revisit as a separate idea if it becomes a real requirement.
• Migration path: incremental, in this order:
  1. Arc-ify stored values (Arc<str>/Arc<String> for log text and metric-captured strings) — independent win, and the load-bearing precondition for cheap cross-actor reads in steps 2–3.
  2. Extract metric computation into a separately-callable stage, still within the single actor.
  3. Split into the reader/primary/profile actors described above.

Current architecture (confirmed in code)

• One process, one actor, today. spawn_engine (src/cli/mod.rs:95) calls engine::spawn(...) exactly once per server run. load_active_profile (src/cli/mod.rs:105-124) loads one active profile's param/metric declarations into that single engine at startup, before ingest starts, with an explicit comment explaining why: "there's no way to reprocess earlier lines later, so this must happen before spawn_ingest" (src/cli/mod.rs:104-106). All other engine::spawn call sites (src/api/mod.rs:853,1459, several in src/config/profile.rs) are test harnesses, not a second production engine instance.
• The actor loop is run(), src/engine/handle.rs:130-365 — a single while let Some(message) = rx.recv().await over one mpsc::Receiver<EngineMessage> (:141). Every mutation (Ingest, IngestMetricSample, DeclareParam/RemoveParamDeclaration, ClearParams, DeclareMetric/UpdateMetric/ RemoveMetricDeclaration, SetMaxLogs, EnsureSortHelperIndex, Barrier — full list src/engine/message.rs:30-63) funnels through this one channel and one task.
• Queries already bypass this actor. EngineQuery (src/engine/message.rs:12-27) doesn't go through the mpsc channel — EngineHandle::ask reads directly off a left_right IndexReader/ReadHandle (src/engine/handle.rs:49-73, entry point src/api/mod.rs:218-219, evaluated by handle_query_ro, src/engine/snapshot/mod.rs:856-858). So read-side scaling is already decoupled from the actor loop today — this idea is entirely about the write/mutation path.
• Per-line ingest (EngineMessage::Ingest arm, src/engine/handle.rs:143-219) does, synchronously, in one left_right write op, per line: ingest-lag/watermark bookkeeping, log-stream header detection, storing the line as DataValue::LogLiteral, auto-param detection (auto_declare_params), running every declared metric's regex against the line (apply_all_metrics → apply_metric_to_log/resolve_param_value, src/engine/snapshot/mod.rs:409-460,821-851), sort-helper-index maintenance, log eviction, and then — after publish — alerts::evaluate_alerts against the resulting snapshot (handle.rs:199-219), which reads metric buckets that were just computed in the same message.
• Module size: src/engine/ is ~5.5k lines excluding tests (snapshot/ mod.rs alone is 1100 lines — the whole EngineSnapshot impl per its own header comment).
• No Arc, everywhere owned data. The index is one flat BTreeMap<IndexKey, DataValue> (snapshot/mod.rs:43) of owned Strings (DataValue::LogLiteral { text: String }, src/engine/data_value.rs:81). Per ingested line, the raw text is cloned at least twice for storage alone (handle.rs:175,184-186), and every metric that matches the line clones its own copy of the label/value strings it captures (apply_metric_to_log, mod.rs:426,439,444) — with N metrics referencing overlapping params, one line produces N independent owned-String allocations of essentially the same substrings.
• left_right's reconciliation deep-clones everything. After a batch of writes, left_right::Absorb::sync_with clones the entire EngineSnapshot (*self = first.clone(), handle.rs:29-31) to reconcile the read-side copy — i.e. the "values should be Arc" note isn't just about avoiding per-line clones, it's also about avoiding a full-BTreeMap-of-Strings deep clone on every reconciliation.
• No text/regex index exists. CompiledFilter::Regex(re) => re.is_match(text) (snapshot/filter.rs:159) runs per line at query time; IndexKey (src/engine/index_key.rs:8-32) has no inverted/full-text structure. Every filter-bar query re-scans and re-matches the whole retained range.
• Backfill is a full synchronous history replay today, in the same actor. declare_metric/update_metric (snapshot/mod.rs:330-391) → backfill_metric (:481-509) collects all retained lines into a Vec and replays apply_metric_to_log over every one, inside the single actor's message handling — blocking ingest/alerts/other declarations for the duration. load_into_engine (src/config/profile.rs:332-384) does this once per metric, awaiting each response, so loading a profile with N metrics against L retained lines costs O(N·L) synchronous scan time. This is also the mechanism 0004-profile-switch-reindexing-cost describes for runtime profile switching via the activate_profile API handler (src/api/mod.rs:712-768, which removes then re-declares via the same messages) — worth reconciling that this research pass separately characterized profile activation at startup as one-shot ("no way to reprocess earlier lines later"), while the earlier research for ticket 099/idea 0004 confirmed a runtime reload path exists via activate_profile. Both are true: startup load is one-shot by necessity (ingest hasn't started yet), and the runtime reload path reuses the same declare/backfill machinery after the fact — but this is worth double-checking before design work starts, since the two research passes framed it differently.
• Alerts are coupled to the single actor's local state. Alert evaluation runs directly in the ingest arm against the just-mutated snapshot (handle.rs:199-211, alerts::evaluate_alerts, src/engine/alerts.rs:211), reading alert_states: HashMap<String, AlertRuntimeState> that lives on the single actor task's own stack frame (handle.rs:137) — there's no mechanism today for one actor to read another actor's in-memory state.
• Existing multi-task precedent is producer-side only. src/engine/ingest.rs's stdin/file-tailing and Prometheus-polling tasks each hold a cloned EngineHandle and push into the same actor's mpsc channel (spawned at src/cli/mod.rs:136-145) — a plausible pattern for "many producers, one consumer," but there's no existing precedent for multiple consuming actors each owning their own left_right write side that need to stay coherent with each other or with a shared upstream.

Implications and things to consider

Performance

• The stated goal — "the primary worker should only parse logs to extract params" — would remove per-line metric-regex evaluation (apply_all_metrics) from the primary's critical path, which is real work today (N metrics × regex match per line). That's a genuine win if profile workers can consume the primary's output without re-paying an equivalent cost themselves — but since a profile worker still has to run its own metrics' regexes against each line to build its own buckets, the total CPU work done across the process doesn't shrink; what changes is that it's no longer all serialized on one task, so it can run concurrently with the primary (a real win on a multi-core box) and a slow/backfilling profile worker no longer blocks raw ingestion.
• The "values in the primary index should be Arcs" change is independently valuable regardless of whether the actor split happens — it directly addresses the confirmed double/triple string cloning per line (storage clone + per-metric label/value clones) and the full-snapshot deep-clone on every left_right reconciliation. Worth sequencing as a prerequisite: switching to Arc<str> (or Arc<String>) for stored text means a profile worker reading "the primary log sequence" can cheaply hold references to the same underlying bytes instead of needing its own copies — this is likely a load-bearing precondition for the split being worth it at all, not a nice-to-have alongside it.
• A profile worker needing "read access to the primary log sequence and derived params index" implies either (a) a second left_right reader over the primary's structures (cheap, if values are Arc'd — see above) or (b) a message-replay feed (the primary notifies profile workers of new lines, per the note's "profiles need to be informed of new messages to index"). Option (b) reintroduces a second data copy per profile worker unless the message itself only carries an Arc/index reference rather than owned data — otherwise this recreates the same N-copies-per-line problem this idea is trying to remove, just distributed across processes/tasks instead of within one.
• Backfill cost (O(N metrics × L retained lines), see 0004-profile-switch-reindexing-cost) doesn't disappear with a split — it moves to whichever worker owns the profile. The upside is it no longer blocks the primary's ingest/query-serving while it happens, which is the main perf argument for doing this at all, per that idea's "Performance" section.
• Full-text/regex indexing on the primary (per the note's "Regex indexing?") is a materially different performance investment from the Arc/split work — an inverted or trigram index trades ingest-time CPU and memory for query-time speed on filter-bar searches, and is worth scoping as its own effort (see 0005-additional-read-indexes) rather than folded into this split, since it doesn't depend on the actor topology.

Code structure

• This is the single largest structural change on the idea list — src/engine/ (~5.5k lines) currently assumes one writer owning one left_right pair. Splitting introduces: a defined "primary perspective" (raw logs + derived/auto params only, no profile-specific metrics) as its own bounded module, a "profile perspective" abstraction for the one active profile, a notification/subscription mechanism from primary to the profile actor (the note's "profiles need to be informed of new messages to index" — doesn't exist in any form today; the closest analogue, ingest.rs's producer tasks feeding one consumer, runs the opposite direction). Resolved (see "Resolved decisions"): the primary signals via e.g. a tokio::sync::watch after each left_right sync completes, and the profile actor re-reads through its own left_right reader rather than the primary pushing data. A query-fan-out/merge layer is still needed for any request that needs both raw logs and profile-derived data in one response (today Logs/Timeline queries read both from a single EngineSnapshot in one pass, e.g. query.rs:154's log_entry_params call against compiled_params from the same snapshot — splitting the data sources means this becomes a real join across two snapshots/workers instead of one borrow).
• Alert evaluation (handle.rs:199-211) currently depends on running immediately after metric bucketing in the same message, against the same snapshot. Resolved (see "Resolved decisions"): alerts move with metric computation into the profile actor, since alert rules are profile-scoped and read the metric buckets computed there.
• EngineHandle's current API (handle.rs:49-73 for reads, the mpsc EngineMessage send for writes) would need to become two APIs, or one handle type parameterized by "primary" vs. "profile" — worth deciding whether api.rs's handlers (which currently hold one EngineHandle in ApiState) address a single combined handle that internally fans out, or explicitly juggle a primary handle plus an active-profile handle.
• Given CLAUDE.md's existing note that engine.rs is "the file to read first" and already the largest/highest-churn file in the repo (13 commits/90d per the repowise summary), this split is also a legibility opportunity, not just a performance one — today a reader has to understand ingest, metrics, alerts, and backfill as one intertwined concern to understand any one of them.

Functionality

• Per-profile message format — resolved, see "Resolved decisions" above: there's no per-profile parsing fork on the primary. Log-stream header detection and auto-param extraction stay a one-time, primary-side, profile-agnostic concern (derive_log_stream_params_from_row, auto_declare_params); any further message-shape interpretation the profile actor needs happens independently, on top of the shared raw Arc text, in the profile actor itself.
• System param suppressions — dropped from scope (see "Resolved decisions"). Revisit as a separate idea if it becomes a real need.
• Concurrency — resolved: still strictly one active profile at a time. The split changes how that one profile's data is computed relative to the primary, not the user-facing model — no multi-profile concurrent indexing.

Not yet considered

• Failure/restart semantics — partially resolved: the profile actor falling behind the primary during a backfill is expected and accepted (see "Resolved decisions" — eviction wins over lag, silent gaps for the evicted range). Still open: what happens if the profile actor crashes outright (not just lags) — does it restart and replay from whatever the primary currently retains, or is there a persisted checkpoint? Not yet decided; likely fine to punt until the split is actually implemented, since "restart and rebuild from currently-retained history" is a reasonable default consistent with the accepted-gaps decision above.
• Retention/eviction coordination — resolved, see "Resolved decisions": no low-water-mark. The primary evicts on its own schedule (evict_excess_logs, handle.rs:195, SetMaxLogs) regardless of the profile actor's progress; a lagging profile actor silently misses data for the evicted range rather than blocking primary eviction.
• Testing story: src/engine/tests.rs (2348 lines) and the test_handle pattern (see ticket 099) are built entirely around one actor with one channel — a split roughly doubles the kinds of integration tests needed (primary alone, profile worker alone, primary+profile interaction) and the existing test harness would need real rework, not just extension. Still open — no resolution yet.
• Migration path — resolved, see "Resolved decisions": incremental, in three steps (Arc-ify stored values → extract metric computation into a still-single-actor stage → split into reader/primary/profile actors).
• Whether this is worth it at all without profiling evidence — resolved, see "Resolved decisions": proceeding without a samply profile, on the code-reading evidence already documented in this file.