Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
98 lines
5.4 KiB
Markdown
98 lines
5.4 KiB
Markdown
# 057 — Line chart for a rate() over sparse/bursty occurrences collapses its x-axis to the data's own tiny time span
|
|
|
|
## Summary
|
|
|
|
Reported symptom: the dashboard line chart for `sum by (code) (rate(KeyPressed[5m]))`
|
|
looks bad/unreadable against a real capture
|
|
(`ottyl --file .../sim-world.log`, a ~10-hour log where `KeyPressed` is a
|
|
user-declared metric matching `msg="key_event"` lines, labeled by
|
|
`code`/`kind`/`msg` — see the profile's metric declaration in
|
|
`~/Library/Application Support/ottyl/settings.json`). In the sample log
|
|
all 20 `key_event` lines (16 `Left`, 3 `Up`, 1 `Down`, all `kind="Press"`)
|
|
occur within the first ~10 seconds of ingestion; there are zero further
|
|
key presses across the remaining ~9h55m of the log.
|
|
|
|
Two compounding causes, both reproducible from this shape of data:
|
|
|
|
1. **`rate()` never emits a trailing zero once occurrences stop.**
|
|
`crate::rate::windowed_rate_series` (`src/rate.rs:40-45`) evaluates
|
|
`rate_at` only *at each existing point's own timestamp* — there is no
|
|
synthetic point emitted at the query's `end_ms` (or at "now") once the
|
|
trailing window has aged out. `evaluate_promql`'s `Sum`/`Avg` branch
|
|
(`src/engine.rs:1621-1648`) calls this per group then merges with
|
|
`merge_series_locf` (`src/engine.rs:1508-1529`), which sweeps only the
|
|
*union of each sub-series' own timestamps* — again nothing beyond the
|
|
last real occurrence. So a series for `code="Left"` literally ends at
|
|
the timestamp of the last `Left` press; there is no data point saying
|
|
"rate is now 0" for the following ~9h55m.
|
|
2. **The frontend's line chart x-axis auto-scales to whatever timestamps
|
|
happen to be in the returned series, not to the queried time range.**
|
|
`PanelChart.tsx`'s `toLineData` (`webapp/src/components/PanelChart.tsx:60-77`)
|
|
builds `rows` only from timestamps present in `series[].points`; the
|
|
`<XAxis dataKey="timestamp_ms" .../>` (`PanelChart.tsx:105-110`) has no
|
|
explicit `domain`, so Recharts defaults to `['auto','auto']` computed
|
|
from `rows` alone. Combined with (1), a panel queried over "last 6
|
|
hours" (or any range spanning the whole log) that only has data in the
|
|
first 10 seconds silently renders an x-axis spanning just those 10
|
|
seconds — not the selected range — with no visual indication the axis
|
|
has been narrowed or that activity stopped. The result is a cramped,
|
|
jagged little chart (peak rate ≈ 16 presses / 300s ≈ 0.053/s, so also
|
|
visually flat/near-zero) that looks broken rather than showing "a
|
|
burst, then nothing."
|
|
|
|
The backend range filter (`start_ms`/`end_ms` in `evaluate_promql`,
|
|
`src/engine.rs:1565-1584`) is applied correctly to which raw occurrences
|
|
are considered — this is not a filtering bug, it's a rendering/shape
|
|
problem in what gets returned and how it's charted.
|
|
|
|
## Scope
|
|
|
|
- Decide and implement a way for the charted line to reflect that the
|
|
rate has decayed to (and stayed at) zero, rather than simply stopping.
|
|
Options to weigh (pick one, document the choice in the PR):
|
|
- Emit a synthetic trailing point at `end_ms` (or "now" for a live
|
|
range) per group whenever the last real point's window has fully
|
|
aged out and its rate would read 0 — cheapest fix, keeps all
|
|
resampling logic where it already lives in `evaluate_promql`.
|
|
- Resample `rate()`/`sum(rate())`/`avg(rate())` series onto a regular
|
|
step grid across `[start_ms, end_ms]` (à la Prometheus's `step`)
|
|
instead of only at occurrence timestamps — bigger change, but fixes
|
|
the same problem for gauges/counters with irregular sampling more
|
|
generally; note `merge_series_windowed` (`src/engine.rs:1484-1503`)
|
|
already buckets on a fixed `MERGE_WINDOW_MS` grid for the
|
|
non-rate `sum()`/`avg()` path, which is closer to this shape already.
|
|
- Whichever approach is chosen, make `PanelChart.tsx`'s line chart honor
|
|
the panel's actual queried range for the x-axis domain (pass
|
|
`range.startMs`/`range.endMs` down from `Dashboards.tsx`'s
|
|
`PanelPreview` into `PanelChart` and set `<XAxis domain={[startMs,
|
|
endMs]} type="number" .../>`, or equivalent) so a chart with sparse
|
|
data doesn't silently narrow the visible window — this is worth doing
|
|
regardless of which backend option above is chosen, since it's the
|
|
more directly misleading half of the bug.
|
|
- Add a fixture reproducing this shape (a metric with occurrences
|
|
clustered in a short burst near the start of a much longer ingested
|
|
window) and a test on `evaluate_promql`/`windowed_rate_series` (or
|
|
wherever the chosen fix lives) asserting the returned series covers
|
|
the full requested range, not just the burst.
|
|
- Out of scope: changing how `KeyPressed`-style occurrence metrics are
|
|
declared, or adding a dedicated "gauge decay" concept — this is about
|
|
the query range/x-axis mismatch specifically.
|
|
|
|
## Acceptance criteria
|
|
|
|
- Querying `sum by (code) (rate(KeyPressed[5m]))` (or an equivalent
|
|
fixture) over a time range much wider than the span of actual
|
|
occurrences returns a series (or is charted) that visibly reflects
|
|
rate-zero for the remainder of the range, rather than a line that
|
|
simply stops.
|
|
- `PanelChart`'s line chart x-axis reflects the panel's selected/queried
|
|
time range, not just the min/max timestamp actually present in the
|
|
returned series — verified by a case where returned data covers only
|
|
a fraction of the selected range.
|
|
- Existing `evaluate_promql`/`rate.rs`/`PanelChart` tests for
|
|
regularly-sampled or continuously-occurring metrics are unaffected.
|
|
|
|
## Depends on
|
|
|
|
None.
|