318 lines
18 KiB
Markdown
318 lines
18 KiB
Markdown
# Idea: split the engine's single actor into a log-ingestion worker + per-profile worker(s)
|
||
|
||
## Original note
|
||
|
||
> This needs to be fleshed out with ai.
|
||
>
|
||
> The general idea is that the primary engine ingesting worker is
|
||
> doing too much. It should only parse logs to extract params.
|
||
>
|
||
> Since profiles carry additional (non-derived) params, metrics,
|
||
> dashboards, and panels - those should be stored separately and can
|
||
> be handled by a different worker. The worker would have read access
|
||
> to the primary log sequence and derived params index, but would
|
||
> construct it's own profile perspective. It has write access and
|
||
> then
|
||
>
|
||
> The profiles need to be informed of new messages to index.
|
||
>
|
||
> What if message needs to be different format? Thats per-profile.
|
||
>
|
||
> System param suppressions?
|
||
>
|
||
> Regex indexing? We probably want text indexing on the primary while
|
||
> also
|
||
>
|
||
> Values in the primary index should be arcs, to prevent cloning
|
||
> allocations.
|
||
|
||
## Resolved decisions (2026-09-07)
|
||
|
||
Clarified interactively; supersedes the open questions raised below where
|
||
they conflict.
|
||
|
||
- **No profiling gate.** Proceeding on the code-reading evidence already in
|
||
this doc (double/triple string clones per line, full-`EngineSnapshot`
|
||
deep-clone on every `left_right` reconciliation) rather than waiting on a
|
||
`samply` profile first.
|
||
- **Three actors, not two.** The split is: **reader actor** (tails
|
||
stdin/file, splits into lines, timestamps, wraps each line in an `Arc`,
|
||
sends downstream) → **primary actor** (ingests timestamped `Arc` lines,
|
||
extracts declared + auto params only, maintains the `param=value` index —
|
||
no metrics, no profile logic) → **profile actor** (holds its own
|
||
`left_right` reader over the primary's index; builds metrics, custom
|
||
params, timeseries, and PromQL evaluation for the one active profile).
|
||
This resolves the "per-profile message format" question below: there is
|
||
no per-profile parsing fork on the primary — format/shape parsing of an
|
||
already-classified line is entirely the profile actor's concern, on top
|
||
of the shared raw text.
|
||
- **Still exactly one active profile at a time.** No concurrent
|
||
multi-profile indexing; the split changes *how* the one active profile's
|
||
data is computed relative to the primary, not the user-facing
|
||
single-active-profile model.
|
||
- **Alerts move to the profile actor.** Alert rules are profile-scoped and
|
||
read metric buckets, so they move with metric computation rather than
|
||
staying on the primary.
|
||
- **Primary→profile signaling:** the primary actor notifies subscribers
|
||
(e.g. a `tokio::sync::watch` carrying the latest processed
|
||
position/index) right after each `left_right` sync completes. The
|
||
profile actor wakes on that signal and re-reads whatever's newly visible
|
||
through its own `left_right` reader — no busy-polling, no second copy of
|
||
the data in transit.
|
||
- **Eviction wins over profile-actor lag.** If the profile actor is
|
||
behind (e.g. mid-backfill on a newly declared metric) when the primary
|
||
evicts old lines (`SetMaxLogs`), the primary evicts on schedule
|
||
regardless — the profile actor silently misses data for the evicted
|
||
range. Simplest option, matches today's single-actor retention model.
|
||
- **System param suppressions: dropped from scope.** Not needed right now;
|
||
revisit as a separate idea if it becomes a real requirement.
|
||
- **Migration path: incremental**, in this order:
|
||
1. Arc-ify stored values (`Arc<str>`/`Arc<String>` for log text and
|
||
metric-captured strings) — independent win, and the load-bearing
|
||
precondition for cheap cross-actor reads in steps 2–3.
|
||
2. Extract metric computation into a separately-callable stage, still
|
||
within the single actor.
|
||
3. Split into the reader/primary/profile actors described above.
|
||
|
||
## Current architecture (confirmed in code)
|
||
|
||
- **One process, one actor, today.** `spawn_engine`
|
||
(`src/cli/mod.rs:95`) calls `engine::spawn(...)` exactly once per
|
||
server run. `load_active_profile` (`src/cli/mod.rs:105-124`) loads
|
||
*one* active profile's param/metric declarations into that single
|
||
engine at startup, before ingest starts, with an explicit comment
|
||
explaining why: "there's no way to reprocess earlier lines later, so
|
||
this must happen before `spawn_ingest`" (`src/cli/mod.rs:104-106`).
|
||
All other `engine::spawn` call sites (`src/api/mod.rs:853,1459`,
|
||
several in `src/config/profile.rs`) are test harnesses, not a second
|
||
production engine instance.
|
||
- The actor loop is `run()`, `src/engine/handle.rs:130-365` — a single
|
||
`while let Some(message) = rx.recv().await` over one
|
||
`mpsc::Receiver<EngineMessage>` (`:141`). Every mutation (`Ingest`,
|
||
`IngestMetricSample`, `DeclareParam`/`RemoveParamDeclaration`,
|
||
`ClearParams`, `DeclareMetric`/`UpdateMetric`/
|
||
`RemoveMetricDeclaration`, `SetMaxLogs`, `EnsureSortHelperIndex`,
|
||
`Barrier` — full list `src/engine/message.rs:30-63`) funnels through
|
||
this one channel and one task.
|
||
- **Queries already bypass this actor.** `EngineQuery`
|
||
(`src/engine/message.rs:12-27`) doesn't go through the mpsc channel
|
||
— `EngineHandle::ask` reads directly off a `left_right`
|
||
`IndexReader`/`ReadHandle` (`src/engine/handle.rs:49-73`, entry point
|
||
`src/api/mod.rs:218-219`, evaluated by `handle_query_ro`,
|
||
`src/engine/snapshot/mod.rs:856-858`). So read-side scaling is
|
||
already decoupled from the actor loop today — this idea is entirely
|
||
about the **write/mutation** path.
|
||
- Per-line ingest (`EngineMessage::Ingest` arm,
|
||
`src/engine/handle.rs:143-219`) does, synchronously, in one
|
||
`left_right` write op, per line: ingest-lag/watermark bookkeeping,
|
||
log-stream header detection, storing the line as
|
||
`DataValue::LogLiteral`, auto-param detection
|
||
(`auto_declare_params`), running **every declared metric's regex**
|
||
against the line (`apply_all_metrics` →
|
||
`apply_metric_to_log`/`resolve_param_value`,
|
||
`src/engine/snapshot/mod.rs:409-460,821-851`), sort-helper-index
|
||
maintenance, log eviction, and then — after publish —
|
||
`alerts::evaluate_alerts` against the resulting snapshot
|
||
(`handle.rs:199-219`), which reads metric buckets that were just
|
||
computed in the same message.
|
||
- Module size: `src/engine/` is ~5.5k lines excluding tests (`snapshot/
|
||
mod.rs` alone is 1100 lines — the whole `EngineSnapshot` impl per its
|
||
own header comment).
|
||
- **No `Arc`, everywhere owned data.** The index is one flat
|
||
`BTreeMap<IndexKey, DataValue>` (`snapshot/mod.rs:43`) of owned
|
||
`String`s (`DataValue::LogLiteral { text: String }`,
|
||
`src/engine/data_value.rs:81`). Per ingested line, the raw text is
|
||
cloned at least twice for storage alone (`handle.rs:175,184-186`),
|
||
and every metric that matches the line clones its own copy of the
|
||
label/value strings it captures (`apply_metric_to_log`,
|
||
`mod.rs:426,439,444`) — with N metrics referencing overlapping
|
||
params, one line produces N independent owned-String allocations of
|
||
essentially the same substrings.
|
||
- **`left_right`'s reconciliation deep-clones everything.** After a
|
||
batch of writes, `left_right::Absorb::sync_with` clones the *entire*
|
||
`EngineSnapshot` (`*self = first.clone()`, `handle.rs:29-31`) to
|
||
reconcile the read-side copy — i.e. the "values should be Arc" note
|
||
isn't just about avoiding per-line clones, it's also about avoiding
|
||
a full-`BTreeMap`-of-Strings deep clone on every reconciliation.
|
||
- **No text/regex index exists.** `CompiledFilter::Regex(re) =>
|
||
re.is_match(text)` (`snapshot/filter.rs:159`) runs per line at query
|
||
time; `IndexKey` (`src/engine/index_key.rs:8-32`) has no
|
||
inverted/full-text structure. Every filter-bar query re-scans and
|
||
re-matches the whole retained range.
|
||
- **Backfill is a full synchronous history replay today, in the same
|
||
actor.** `declare_metric`/`update_metric`
|
||
(`snapshot/mod.rs:330-391`) → `backfill_metric` (`:481-509`) collects
|
||
**all** retained lines into a `Vec` and replays
|
||
`apply_metric_to_log` over every one, inside the single actor's
|
||
message handling — blocking ingest/alerts/other declarations for the
|
||
duration. `load_into_engine` (`src/config/profile.rs:332-384`) does
|
||
this once per metric, awaiting each response, so loading a profile
|
||
with N metrics against L retained lines costs O(N·L) synchronous
|
||
scan time. This is also the mechanism [[0004-profile-switch-reindexing-cost]]
|
||
describes for *runtime* profile switching via the `activate_profile`
|
||
API handler (`src/api/mod.rs:712-768`, which removes then
|
||
re-declares via the same messages) — worth reconciling that this
|
||
research pass separately characterized profile activation at
|
||
startup as one-shot ("no way to reprocess earlier lines later"),
|
||
while the earlier research for ticket 099/idea 0004 confirmed a
|
||
*runtime* reload path exists via `activate_profile`. Both are true:
|
||
startup load is one-shot by necessity (ingest hasn't started yet),
|
||
and the runtime reload path reuses the same declare/backfill
|
||
machinery after the fact — but this is worth double-checking before
|
||
design work starts, since the two research passes framed it
|
||
differently.
|
||
- **Alerts are coupled to the single actor's local state.** Alert
|
||
evaluation runs directly in the ingest arm against the just-mutated
|
||
snapshot (`handle.rs:199-211`, `alerts::evaluate_alerts`,
|
||
`src/engine/alerts.rs:211`), reading `alert_states:
|
||
HashMap<String, AlertRuntimeState>` that lives on the single actor
|
||
task's own stack frame (`handle.rs:137`) — there's no mechanism today
|
||
for one actor to read another actor's in-memory state.
|
||
- **Existing multi-task precedent is producer-side only.**
|
||
`src/engine/ingest.rs`'s stdin/file-tailing and Prometheus-polling
|
||
tasks each hold a cloned `EngineHandle` and push into the *same*
|
||
actor's mpsc channel (spawned at `src/cli/mod.rs:136-145`) — a
|
||
plausible pattern for "many producers, one consumer," but there's no
|
||
existing precedent for multiple *consuming* actors each owning their
|
||
own `left_right` write side that need to stay coherent with each
|
||
other or with a shared upstream.
|
||
|
||
## Implications and things to consider
|
||
|
||
### Performance
|
||
|
||
- The stated goal — "the primary worker should only parse logs to
|
||
extract params" — would remove per-line metric-regex evaluation
|
||
(`apply_all_metrics`) from the primary's critical path, which is
|
||
real work today (N metrics × regex match per line). That's a
|
||
genuine win *if* profile workers can consume the primary's output
|
||
without re-paying an equivalent cost themselves — but since a
|
||
profile worker still has to run its own metrics' regexes against
|
||
each line to build its own buckets, the total CPU work done across
|
||
the process doesn't shrink; what changes is that it's no longer all
|
||
serialized on one task, so it can run concurrently with the primary
|
||
(a real win on a multi-core box) and a slow/backfilling profile
|
||
worker no longer blocks raw ingestion.
|
||
- The "values in the primary index should be Arcs" change is
|
||
independently valuable regardless of whether the actor split
|
||
happens — it directly addresses the confirmed double/triple string
|
||
cloning per line (storage clone + per-metric label/value clones) and
|
||
the full-snapshot deep-clone on every `left_right` reconciliation.
|
||
Worth sequencing as a prerequisite: switching to `Arc<str>` (or
|
||
`Arc<String>`) for stored text means a profile worker reading "the
|
||
primary log sequence" can cheaply hold references to the same
|
||
underlying bytes instead of needing its own copies — this is likely
|
||
a load-bearing precondition for the split being worth it at all, not
|
||
a nice-to-have alongside it.
|
||
- A profile worker needing "read access to the primary log sequence
|
||
and derived params index" implies either (a) a second `left_right`
|
||
reader over the primary's structures (cheap, if values are Arc'd —
|
||
see above) or (b) a message-replay feed (the primary notifies
|
||
profile workers of new lines, per the note's "profiles need to be
|
||
informed of new messages to index"). Option (b) reintroduces a
|
||
second data copy per profile worker unless the message itself only
|
||
carries an `Arc`/index reference rather than owned data — otherwise
|
||
this recreates the same N-copies-per-line problem this idea is
|
||
trying to remove, just distributed across processes/tasks instead of
|
||
within one.
|
||
- Backfill cost (O(N metrics × L retained lines), see [[0004-profile-switch-reindexing-cost]])
|
||
doesn't disappear with a split — it moves to whichever worker owns
|
||
the profile. The upside is it no longer blocks the *primary's*
|
||
ingest/query-serving while it happens, which is the main perf
|
||
argument for doing this at all, per that idea's "Performance"
|
||
section.
|
||
- Full-text/regex indexing on the primary (per the note's "Regex
|
||
indexing?") is a materially different performance investment from
|
||
the Arc/split work — an inverted or trigram index trades ingest-time
|
||
CPU and memory for query-time speed on filter-bar searches, and is
|
||
worth scoping as its own effort (see [[0005-additional-read-indexes]])
|
||
rather than folded into this split, since it doesn't depend on the
|
||
actor topology.
|
||
|
||
### Code structure
|
||
|
||
- This is the single largest structural change on the idea list —
|
||
`src/engine/` (~5.5k lines) currently assumes one writer owning one
|
||
`left_right` pair. Splitting introduces: a defined "primary
|
||
perspective" (raw logs + derived/auto params only, no
|
||
profile-specific metrics) as its own bounded module, a
|
||
"profile perspective" abstraction for the one active profile, a
|
||
notification/subscription mechanism from primary to the profile actor
|
||
(the note's "profiles need to be informed of new messages to index" —
|
||
doesn't exist in any form today; the closest analogue, `ingest.rs`'s
|
||
producer tasks feeding one consumer, runs the opposite direction).
|
||
**Resolved** (see "Resolved decisions"): the primary signals via e.g. a
|
||
`tokio::sync::watch` after each `left_right` sync completes, and the
|
||
profile actor re-reads through its own `left_right` reader rather than
|
||
the primary pushing data. A query-fan-out/merge layer is still needed
|
||
for any request
|
||
that needs both raw logs and profile-derived data in one response
|
||
(today `Logs`/`Timeline` queries read both from a single
|
||
`EngineSnapshot` in one pass, e.g. `query.rs:154`'s
|
||
`log_entry_params` call against `compiled_params` from the same
|
||
snapshot — splitting the data sources means this becomes a real join
|
||
across two snapshots/workers instead of one borrow).
|
||
- Alert evaluation (`handle.rs:199-211`) currently depends on running
|
||
immediately after metric bucketing in the same message, against the
|
||
same snapshot. **Resolved** (see "Resolved decisions"): alerts move
|
||
with metric computation into the profile actor, since alert rules are
|
||
profile-scoped and read the metric buckets computed there.
|
||
- `EngineHandle`'s current API (`handle.rs:49-73` for reads,
|
||
the mpsc `EngineMessage` send for writes) would need to become two
|
||
APIs, or one handle type parameterized by "primary" vs. "profile" —
|
||
worth deciding whether `api.rs`'s handlers (which currently hold one
|
||
`EngineHandle` in `ApiState`) address a single combined handle that
|
||
internally fans out, or explicitly juggle a primary handle plus an
|
||
active-profile handle.
|
||
- Given `CLAUDE.md`'s existing note that `engine.rs` is "the file to
|
||
read first" and already the largest/highest-churn file in the repo
|
||
(13 commits/90d per the repowise summary), this split is also a
|
||
legibility opportunity, not just a performance one — today a reader
|
||
has to understand ingest, metrics, alerts, and backfill as one
|
||
intertwined concern to understand any one of them.
|
||
|
||
### Functionality
|
||
|
||
- **Per-profile message format** — **resolved**, see "Resolved decisions"
|
||
above: there's no per-profile parsing fork on the primary. Log-stream
|
||
header detection and auto-param extraction stay a one-time, primary-side,
|
||
profile-agnostic concern (`derive_log_stream_params_from_row`,
|
||
`auto_declare_params`); any further message-shape interpretation the
|
||
profile actor needs happens independently, on top of the shared raw
|
||
`Arc` text, in the profile actor itself.
|
||
- **System param suppressions** — **dropped from scope** (see "Resolved
|
||
decisions"). Revisit as a separate idea if it becomes a real need.
|
||
- **Concurrency** — **resolved**: still strictly one active profile at a
|
||
time. The split changes *how* that one profile's data is computed
|
||
relative to the primary, not the user-facing model — no multi-profile
|
||
concurrent indexing.
|
||
|
||
### Not yet considered
|
||
|
||
- **Failure/restart semantics** — partially resolved: the profile actor
|
||
falling behind the primary during a backfill is expected and accepted
|
||
(see "Resolved decisions" — eviction wins over lag, silent gaps for the
|
||
evicted range). Still open: what happens if the profile actor *crashes*
|
||
outright (not just lags) — does it restart and replay from whatever the
|
||
primary currently retains, or is there a persisted checkpoint? Not yet
|
||
decided; likely fine to punt until the split is actually implemented,
|
||
since "restart and rebuild from currently-retained history" is a
|
||
reasonable default consistent with the accepted-gaps decision above.
|
||
- **Retention/eviction coordination** — **resolved**, see "Resolved
|
||
decisions": no low-water-mark. The primary evicts on its own schedule
|
||
(`evict_excess_logs`, `handle.rs:195`, `SetMaxLogs`) regardless of the
|
||
profile actor's progress; a lagging profile actor silently misses data
|
||
for the evicted range rather than blocking primary eviction.
|
||
- **Testing story**: `src/engine/tests.rs` (2348 lines) and the
|
||
`test_handle` pattern (see ticket 099) are built entirely around one
|
||
actor with one channel — a split roughly doubles the kinds of
|
||
integration tests needed (primary alone, profile worker alone,
|
||
primary+profile interaction) and the existing test harness would
|
||
need real rework, not just extension. Still open — no resolution yet.
|
||
- **Migration path** — **resolved**, see "Resolved decisions": incremental,
|
||
in three steps (Arc-ify stored values → extract metric computation into
|
||
a still-single-actor stage → split into reader/primary/profile actors).
|
||
- **Whether this is worth it at all without profiling evidence** —
|
||
**resolved**, see "Resolved decisions": proceeding without a `samply`
|
||
profile, on the code-reading evidence already documented in this file.
|