# Idea: dependent/nested param extraction and jq-style JSON extraction ## Original note 1. A param currently auto-detects or is declared via a regex against the raw log line — that's the baseline, already true today. 2. Semantics for "has a value": if the message has a non-empty or missing capture for the target group, treat that as the param having a value (as opposed to today's behavior — needs confirmation, see Open questions). 3. Extract a param from *another param's already-captured string*, applying a second regex on top of it, rather than always re-matching the raw line. 4. When a string param (or the whole message) is JSON, extract fields via `jq` syntax rather than the current plain dot-path. ## Current state (confirmed in code) - `ParamDeclaration` (`src/engine/data_value.rs:6-20`) supports exactly one regex + one named capture group per param, or one JSON dot-path, or the computed `elapsed_since_last` — these three modes are mutually exclusive (validated in `src/engine/snapshot/mod.rs:618-633`). - `extract_param_raw` (`src/engine/snapshot/params.rs:85-99`) *always* operates on the raw ingested line's text — there is no "source param" concept anywhere in `ParamDeclaration`/ `ParamDeclarationRecord`/`ParamDeclarationDto`. Every param independently re-parses the same raw line; none can chain off another param's captured value. - JSON extraction (`extract_json_field`, `src/engine/data_value.rs:22-39`) requires the whole line to parse as a JSON object, and walks a plain dot-separated path through nested objects only — no array indexing, no jq filters/pipes/functions. Frontend surface: `webapp/src/components/Params.tsx:24-33` (gates the "JSON field" option on `looksLikeJsonObject`), free-text path input, no path-builder/autocomplete. - No `jq` crate or jq-like dependency exists in the codebase today. ## Why this is an idea, not a ticket yet This is a real feature addition with several open design questions that change its shape and scope significantly — it's not a well-bounded bug fix or mechanical change. ## Open questions - **Chaining model**: should a param declaration gain an optional "source param" field (extract from another param's value instead of the raw line)? If so, declarations form a small dependency graph — what happens on a cycle, or when the source param has no value for a given line (does the dependent param also have no value, silently)? Does extraction order need to become topological instead of the current independent-per-declaration model? - **Missing/empty capture semantics**: what should currently happen when a regex matches but the named group didn't participate (e.g. an optional group in an alternation)? Worth checking `extract_param_raw`'s exact behavior before assuming this needs a change — it may already do the right thing. - **jq scope**: is full jq syntax the actual goal, or would extending the existing dot-path extractor with array indexing (`items[0].id`) cover most real cases far more cheaply than pulling in a jq implementation (e.g. the `jaq` crate) as a new dependency? jq's filter language (pipes, `select()`, `map()`) is a much bigger surface to expose safely in a declaration UI than a path string. - **UI shape**: `Params.tsx`'s declaration form is built around one regex/one capture group or one JSON path; a chained/jq model needs a materially different UI (e.g. "extract from: [raw message ▾] or [param: session_id ▾]" plus a jq expression box with live preview against a sample line). ## Suggested next step Worth a short design pass (e.g. `ce-brainstorm` or a quick conversation) to settle the chaining model and jq-vs-extended-dot-path question before this gets broken into tickets — the two features (2/3 and 4) could also ship independently rather than as one big change. ## Implications and things to consider ### Performance - Chained extraction means a line can now do N regex/JSON passes instead of 1 per param, and a dependent param's cost is paid *after* its source param already ran — worst case, a profile with a long dependency chain multiplies per-line ingest cost by chain depth. This lands directly in the single actor's per-line ingest path (see [[0001-actor-split]] — if that split happens first, param extraction is exactly the workload it isolates, so it's worth sequencing this after that decision rather than before). - A `jq`-style engine (even a Rust one like `jaq`) is meaningfully more expensive per invocation than a compiled `regex::Regex` capture — evaluating a jq filter against a parsed `serde_json::Value` for *every* ingested line for *every* JSON-extraction param is a different cost profile than today's simple dot-path walk. Worth benchmarking before committing to full jq rather than assuming it's free. - Topological ordering of dependent params (if declarations form a DAG) adds a per-declaration-change cost (recompute order) but not a per-line cost, as long as the order is cached after each add/remove/edit rather than recomputed per line. ### Code structure - `extract_param_raw` (`src/engine/snapshot/params.rs:85-99`) currently has a flat signature `(text: &str, declaration) -> Option`. Chaining requires either passing in the full set of already-extracted param values for the current line (so a declaration can look up its source param's value) or restructuring extraction as a fold over declarations in dependency order — a real shape change to this function and its caller, not a param-by-param tweak. - The "three mutually exclusive modes" validation in `src/engine/snapshot/mod.rs:618-633` needs a fourth mode (or an orthogonal "source" field crossed with the existing regex/JSON modes) — worth deciding whether source-param and JSON-field are independent axes (e.g. "JSON-extract from param X" as well as from the raw line) since that combination is implied by note #3. - This is exactly the shape of change flagged in this repo's `CLAUDE.md` as needing three coordinated updates (engine message + handler, settings/profile persistence, api.rs route) plus a frontend affordance — touches more surface area than the "Current state" section alone suggests. ### Functionality - Cycle detection isn't optional once chaining exists — a user could declare param B from param A and (accidentally, via UI) param A from param B. Needs either a hard validation error at declare-time or a documented "first N declarations in insertion order" fallback. - Auto-derived params (JSON auto-flattening, `flatten_json_object`/`flatten_json_object_into`, `src/engine/data_value.rs:41-60+`) aren't user declarations — decide whether a chained/jq param can source from an *auto-derived* param, which would tie this feature to auto-detection internals that weren't designed to be a stable "source" API. - Backfill: declaring a new dependent param triggers the same `backfill_metric`-style history replay as any param declaration (see [[0004-profile-switch-reindexing-cost]] for the general backfill-cost concern) — a chain of N dependent params means N passes over retained history on declare, not just N passes per new line going forward. ### Not yet considered - Error/partial-failure UX: what does the UI show for a param whose jq expression is syntactically valid but fails on a particular line's JSON shape (e.g. missing key, type mismatch) — silent no-value, or a visible per-line extraction error surfaced somewhere? - Whether jq's more powerful operations (`map`, `select`, arithmetic) should be allowed at all in a param *value* context, where the result is expected to be a single scalar — most of jq's power is about reshaping/filtering collections, which doesn't map cleanly onto "this param has one value per log line." - No investigation yet into how this interacts with `sort_by` (`SortHelperKey`) or PromQL — do chained/jq-derived params get the same downstream treatment (sortable, usable in metric declarations, usable in dashboard queries) as directly-extracted params, or are there gaps once the extraction pipeline is no longer "one regex against the raw line"?