Idea: dependent/nested param extraction and jq-style JSON extraction

Original note

A param currently auto-detects or is declared via a regex against the raw log line — that's the baseline, already true today.
Semantics for "has a value": if the message has a non-empty or missing capture for the target group, treat that as the param having a value (as opposed to today's behavior — needs confirmation, see Open questions).
Extract a param from another param's already-captured string, applying a second regex on top of it, rather than always re-matching the raw line.
When a string param (or the whole message) is JSON, extract fields via jq syntax rather than the current plain dot-path.

Current state (confirmed in code)

ParamDeclaration (src/engine/data_value.rs:6-20) supports exactly one regex + one named capture group per param, or one JSON dot-path, or the computed elapsed_since_last — these three modes are mutually exclusive (validated in src/engine/snapshot/mod.rs:618-633).
extract_param_raw (src/engine/snapshot/params.rs:85-99) always operates on the raw ingested line's text — there is no "source param" concept anywhere in ParamDeclaration/ ParamDeclarationRecord/ParamDeclarationDto. Every param independently re-parses the same raw line; none can chain off another param's captured value.
JSON extraction (extract_json_field, src/engine/data_value.rs:22-39) requires the whole line to parse as a JSON object, and walks a plain dot-separated path through nested objects only — no array indexing, no jq filters/pipes/functions. Frontend surface: webapp/src/components/Params.tsx:24-33 (gates the "JSON field" option on looksLikeJsonObject), free-text path input, no path-builder/autocomplete.
No jq crate or jq-like dependency exists in the codebase today.

Why this is an idea, not a ticket yet

This is a real feature addition with several open design questions that change its shape and scope significantly — it's not a well-bounded bug fix or mechanical change.

Open questions

Chaining model: should a param declaration gain an optional "source param" field (extract from another param's value instead of the raw line)? If so, declarations form a small dependency graph — what happens on a cycle, or when the source param has no value for a given line (does the dependent param also have no value, silently)? Does extraction order need to become topological instead of the current independent-per-declaration model?
Missing/empty capture semantics: what should currently happen when a regex matches but the named group didn't participate (e.g. an optional group in an alternation)? Worth checking extract_param_raw's exact behavior before assuming this needs a change — it may already do the right thing.
jq scope: is full jq syntax the actual goal, or would extending the existing dot-path extractor with array indexing (items[0].id) cover most real cases far more cheaply than pulling in a jq implementation (e.g. the jaq crate) as a new dependency? jq's filter language (pipes, select(), map()) is a much bigger surface to expose safely in a declaration UI than a path string.
UI shape: Params.tsx's declaration form is built around one regex/one capture group or one JSON path; a chained/jq model needs a materially different UI (e.g. "extract from: [raw message ▾] or [param: session_id ▾]" plus a jq expression box with live preview against a sample line).

Suggested next step

Worth a short design pass (e.g. ce-brainstorm or a quick conversation) to settle the chaining model and jq-vs-extended-dot-path question before this gets broken into tickets — the two features (2/3 and 4) could also ship independently rather than as one big change.

Implications and things to consider

Performance

Chained extraction means a line can now do N regex/JSON passes instead of 1 per param, and a dependent param's cost is paid after its source param already ran — worst case, a profile with a long dependency chain multiplies per-line ingest cost by chain depth. This lands directly in the single actor's per-line ingest path (see 0001-actor-split — if that split happens first, param extraction is exactly the workload it isolates, so it's worth sequencing this after that decision rather than before).
A jq-style engine (even a Rust one like jaq) is meaningfully more expensive per invocation than a compiled regex::Regex capture — evaluating a jq filter against a parsed serde_json::Value for every ingested line for every JSON-extraction param is a different cost profile than today's simple dot-path walk. Worth benchmarking before committing to full jq rather than assuming it's free.
Topological ordering of dependent params (if declarations form a DAG) adds a per-declaration-change cost (recompute order) but not a per-line cost, as long as the order is cached after each add/remove/edit rather than recomputed per line.

Code structure

extract_param_raw (src/engine/snapshot/params.rs:85-99) currently has a flat signature (text: &str, declaration) -> Option<String>. Chaining requires either passing in the full set of already-extracted param values for the current line (so a declaration can look up its source param's value) or restructuring extraction as a fold over declarations in dependency order — a real shape change to this function and its caller, not a param-by-param tweak.
The "three mutually exclusive modes" validation in src/engine/snapshot/mod.rs:618-633 needs a fourth mode (or an orthogonal "source" field crossed with the existing regex/JSON modes) — worth deciding whether source-param and JSON-field are independent axes (e.g. "JSON-extract from param X" as well as from the raw line) since that combination is implied by note #3.
This is exactly the shape of change flagged in this repo's CLAUDE.md as needing three coordinated updates (engine message + handler, settings/profile persistence, api.rs route) plus a frontend affordance — touches more surface area than the "Current state" section alone suggests.

Functionality

Cycle detection isn't optional once chaining exists — a user could declare param B from param A and (accidentally, via UI) param A from param B. Needs either a hard validation error at declare-time or a documented "first N declarations in insertion order" fallback.
Auto-derived params (JSON auto-flattening, flatten_json_object/flatten_json_object_into, src/engine/data_value.rs:41-60+) aren't user declarations — decide whether a chained/jq param can source from an auto-derived param, which would tie this feature to auto-detection internals that weren't designed to be a stable "source" API.
Backfill: declaring a new dependent param triggers the same backfill_metric-style history replay as any param declaration (see 0004-profile-switch-reindexing-cost for the general backfill-cost concern) — a chain of N dependent params means N passes over retained history on declare, not just N passes per new line going forward.

Not yet considered

Error/partial-failure UX: what does the UI show for a param whose jq expression is syntactically valid but fails on a particular line's JSON shape (e.g. missing key, type mismatch) — silent no-value, or a visible per-line extraction error surfaced somewhere?
Whether jq's more powerful operations (map, select, arithmetic) should be allowed at all in a param value context, where the result is expected to be a single scalar — most of jq's power is about reshaping/filtering collections, which doesn't map cleanly onto "this param has one value per log line."
No investigation yet into how this interacts with sort_by (SortHelperKey) or PromQL — do chained/jq-derived params get the same downstream treatment (sortable, usable in metric declarations, usable in dashboard queries) as directly-extracted params, or are there gaps once the extraction pipeline is no longer "one regex against the raw line"?

8.1 KiB Raw Blame History