ottyl/ideas/0002-nested-and-jq-param-extraction.md

# Idea: dependent/nested param extraction and jq-style JSON extraction

## Original note

1. A param currently auto-detects or is declared via a regex against
   the raw log line — that's the baseline, already true today.
2. Semantics for "has a value": if the message has a non-empty or
   missing capture for the target group, treat that as the param
   having a value (as opposed to today's behavior — needs
   confirmation, see Open questions).
3. Extract a param from *another param's already-captured string*,
   applying a second regex on top of it, rather than always
   re-matching the raw line.
4. When a string param (or the whole message) is JSON, extract fields
   via `jq` syntax rather than the current plain dot-path.

## Current state (confirmed in code)

- `ParamDeclaration` (`src/engine/data_value.rs:6-20`) supports exactly
  one regex + one named capture group per param, or one JSON
  dot-path, or the computed `elapsed_since_last` — these three modes
  are mutually exclusive (validated in
  `src/engine/snapshot/mod.rs:618-633`).
- `extract_param_raw` (`src/engine/snapshot/params.rs:85-99`) *always*
  operates on the raw ingested line's text — there is no "source
  param" concept anywhere in `ParamDeclaration`/
  `ParamDeclarationRecord`/`ParamDeclarationDto`. Every param
  independently re-parses the same raw line; none can chain off
  another param's captured value.
- JSON extraction (`extract_json_field`, `src/engine/data_value.rs:22-39`)
  requires the whole line to parse as a JSON object, and walks a plain
  dot-separated path through nested objects only — no array indexing,
  no jq filters/pipes/functions. Frontend surface:
  `webapp/src/components/Params.tsx:24-33` (gates the "JSON field"
  option on `looksLikeJsonObject`), free-text path input, no
  path-builder/autocomplete.
- No `jq` crate or jq-like dependency exists in the codebase today.

## Why this is an idea, not a ticket yet

This is a real feature addition with several open design questions
that change its shape and scope significantly — it's not a
well-bounded bug fix or mechanical change.

## Open questions

- **Chaining model**: should a param declaration gain an optional
  "source param" field (extract from another param's value instead of
  the raw line)? If so, declarations form a small dependency graph —
  what happens on a cycle, or when the source param has no value for a
  given line (does the dependent param also have no value, silently)?
  Does extraction order need to become topological instead of the
  current independent-per-declaration model?
- **Missing/empty capture semantics**: what should currently happen
  when a regex matches but the named group didn't participate (e.g. an
  optional group in an alternation)? Worth checking
  `extract_param_raw`'s exact behavior before assuming this needs a
  change — it may already do the right thing.
- **jq scope**: is full jq syntax the actual goal, or would extending
  the existing dot-path extractor with array indexing (`items[0].id`)
  cover most real cases far more cheaply than pulling in a jq
  implementation (e.g. the `jaq` crate) as a new dependency? jq's
  filter language (pipes, `select()`, `map()`) is a much bigger surface
  to expose safely in a declaration UI than a path string.
- **UI shape**: `Params.tsx`'s declaration form is built around one
  regex/one capture group or one JSON path; a chained/jq model needs a
  materially different UI (e.g. "extract from: [raw message ▾] or
  [param: session_id ▾]" plus a jq expression box with live preview
  against a sample line).

## Suggested next step

Worth a short design pass (e.g. `ce-brainstorm` or a quick
conversation) to settle the chaining model and jq-vs-extended-dot-path
question before this gets broken into tickets — the two features (2/3
and 4) could also ship independently rather than as one big change.

## Implications and things to consider

### Performance

- Chained extraction means a line can now do N regex/JSON passes
  instead of 1 per param, and a dependent param's cost is paid *after*
  its source param already ran — worst case, a profile with a long
  dependency chain multiplies per-line ingest cost by chain depth.
  This lands directly in the single actor's per-line ingest path (see
  [[0001-actor-split]] — if that split happens first, param extraction
  is exactly the workload it isolates, so it's worth sequencing this
  after that decision rather than before).
- A `jq`-style engine (even a Rust one like `jaq`) is meaningfully
  more expensive per invocation than a compiled `regex::Regex` capture
  — evaluating a jq filter against a parsed `serde_json::Value` for
  *every* ingested line for *every* JSON-extraction param is a
  different cost profile than today's simple dot-path walk. Worth
  benchmarking before committing to full jq rather than assuming it's
  free.
- Topological ordering of dependent params (if declarations form a
  DAG) adds a per-declaration-change cost (recompute order) but not a
  per-line cost, as long as the order is cached after each
  add/remove/edit rather than recomputed per line.

### Code structure

- `extract_param_raw` (`src/engine/snapshot/params.rs:85-99`) currently
  has a flat signature `(text: &str, declaration) -> Option<String>`.
  Chaining requires either passing in the full set of already-extracted
  param values for the current line (so a declaration can look up its
  source param's value) or restructuring extraction as a fold over
  declarations in dependency order — a real shape change to this
  function and its caller, not a param-by-param tweak.
- The "three mutually exclusive modes" validation in
  `src/engine/snapshot/mod.rs:618-633` needs a fourth mode (or an
  orthogonal "source" field crossed with the existing regex/JSON
  modes) — worth deciding whether source-param and JSON-field are
  independent axes (e.g. "JSON-extract from param X" as well as from
  the raw line) since that combination is implied by note #3.
- This is exactly the shape of change flagged in this repo's
  `CLAUDE.md` as needing three coordinated updates (engine message +
  handler, settings/profile persistence, api.rs route) plus a frontend
  affordance — touches more surface area than the "Current state"
  section alone suggests.

### Functionality

- Cycle detection isn't optional once chaining exists — a user could
  declare param B from param A and (accidentally, via UI) param A from
  param B. Needs either a hard validation error at declare-time or a
  documented "first N declarations in insertion order" fallback.
- Auto-derived params (JSON auto-flattening,
  `flatten_json_object`/`flatten_json_object_into`,
  `src/engine/data_value.rs:41-60+`) aren't user declarations — decide
  whether a chained/jq param can source from an *auto-derived* param,
  which would tie this feature to auto-detection internals that
  weren't designed to be a stable "source" API.
- Backfill: declaring a new dependent param triggers the same
  `backfill_metric`-style history replay as any param declaration
  (see [[0004-profile-switch-reindexing-cost]] for the general
  backfill-cost concern) — a chain of N dependent params means N
  passes over retained history on declare, not just N passes per new
  line going forward.

### Not yet considered

- Error/partial-failure UX: what does the UI show for a param whose
  jq expression is syntactically valid but fails on a particular
  line's JSON shape (e.g. missing key, type mismatch) — silent
  no-value, or a visible per-line extraction error surfaced somewhere?
- Whether jq's more powerful operations (`map`, `select`, arithmetic)
  should be allowed at all in a param *value* context, where the
  result is expected to be a single scalar — most of jq's power is
  about reshaping/filtering collections, which doesn't map cleanly
  onto "this param has one value per log line."
- No investigation yet into how this interacts with `sort_by`
  (`SortHelperKey`) or PromQL — do chained/jq-derived params get the
  same downstream treatment (sortable, usable in metric declarations,
  usable in dashboard queries) as directly-extracted params, or are
  there gaps once the extraction pipeline is no longer "one regex
  against the raw line"?