Add ticket 057: rate() line chart x-axis collapses to sparse occurrence window
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
d17cfe9b83
commit
b15fbcee04
@@ -0,0 +1,97 @@
|
||||
# 057 — Line chart for a rate() over sparse/bursty occurrences collapses its x-axis to the data's own tiny time span
|
||||
|
||||
## Summary
|
||||
|
||||
Reported symptom: the dashboard line chart for `sum by (code) (rate(KeyPressed[5m]))`
|
||||
looks bad/unreadable against a real capture
|
||||
(`ottyl --file .../sim-world.log`, a ~10-hour log where `KeyPressed` is a
|
||||
user-declared metric matching `msg="key_event"` lines, labeled by
|
||||
`code`/`kind`/`msg` — see the profile's metric declaration in
|
||||
`~/Library/Application Support/ottyl/settings.json`). In the sample log
|
||||
all 20 `key_event` lines (16 `Left`, 3 `Up`, 1 `Down`, all `kind="Press"`)
|
||||
occur within the first ~10 seconds of ingestion; there are zero further
|
||||
key presses across the remaining ~9h55m of the log.
|
||||
|
||||
Two compounding causes, both reproducible from this shape of data:
|
||||
|
||||
1. **`rate()` never emits a trailing zero once occurrences stop.**
|
||||
`crate::rate::windowed_rate_series` (`src/rate.rs:40-45`) evaluates
|
||||
`rate_at` only *at each existing point's own timestamp* — there is no
|
||||
synthetic point emitted at the query's `end_ms` (or at "now") once the
|
||||
trailing window has aged out. `evaluate_promql`'s `Sum`/`Avg` branch
|
||||
(`src/engine.rs:1621-1648`) calls this per group then merges with
|
||||
`merge_series_locf` (`src/engine.rs:1508-1529`), which sweeps only the
|
||||
*union of each sub-series' own timestamps* — again nothing beyond the
|
||||
last real occurrence. So a series for `code="Left"` literally ends at
|
||||
the timestamp of the last `Left` press; there is no data point saying
|
||||
"rate is now 0" for the following ~9h55m.
|
||||
2. **The frontend's line chart x-axis auto-scales to whatever timestamps
|
||||
happen to be in the returned series, not to the queried time range.**
|
||||
`PanelChart.tsx`'s `toLineData` (`webapp/src/components/PanelChart.tsx:60-77`)
|
||||
builds `rows` only from timestamps present in `series[].points`; the
|
||||
`<XAxis dataKey="timestamp_ms" .../>` (`PanelChart.tsx:105-110`) has no
|
||||
explicit `domain`, so Recharts defaults to `['auto','auto']` computed
|
||||
from `rows` alone. Combined with (1), a panel queried over "last 6
|
||||
hours" (or any range spanning the whole log) that only has data in the
|
||||
first 10 seconds silently renders an x-axis spanning just those 10
|
||||
seconds — not the selected range — with no visual indication the axis
|
||||
has been narrowed or that activity stopped. The result is a cramped,
|
||||
jagged little chart (peak rate ≈ 16 presses / 300s ≈ 0.053/s, so also
|
||||
visually flat/near-zero) that looks broken rather than showing "a
|
||||
burst, then nothing."
|
||||
|
||||
The backend range filter (`start_ms`/`end_ms` in `evaluate_promql`,
|
||||
`src/engine.rs:1565-1584`) is applied correctly to which raw occurrences
|
||||
are considered — this is not a filtering bug, it's a rendering/shape
|
||||
problem in what gets returned and how it's charted.
|
||||
|
||||
## Scope
|
||||
|
||||
- Decide and implement a way for the charted line to reflect that the
|
||||
rate has decayed to (and stayed at) zero, rather than simply stopping.
|
||||
Options to weigh (pick one, document the choice in the PR):
|
||||
- Emit a synthetic trailing point at `end_ms` (or "now" for a live
|
||||
range) per group whenever the last real point's window has fully
|
||||
aged out and its rate would read 0 — cheapest fix, keeps all
|
||||
resampling logic where it already lives in `evaluate_promql`.
|
||||
- Resample `rate()`/`sum(rate())`/`avg(rate())` series onto a regular
|
||||
step grid across `[start_ms, end_ms]` (à la Prometheus's `step`)
|
||||
instead of only at occurrence timestamps — bigger change, but fixes
|
||||
the same problem for gauges/counters with irregular sampling more
|
||||
generally; note `merge_series_windowed` (`src/engine.rs:1484-1503`)
|
||||
already buckets on a fixed `MERGE_WINDOW_MS` grid for the
|
||||
non-rate `sum()`/`avg()` path, which is closer to this shape already.
|
||||
- Whichever approach is chosen, make `PanelChart.tsx`'s line chart honor
|
||||
the panel's actual queried range for the x-axis domain (pass
|
||||
`range.startMs`/`range.endMs` down from `Dashboards.tsx`'s
|
||||
`PanelPreview` into `PanelChart` and set `<XAxis domain={[startMs,
|
||||
endMs]} type="number" .../>`, or equivalent) so a chart with sparse
|
||||
data doesn't silently narrow the visible window — this is worth doing
|
||||
regardless of which backend option above is chosen, since it's the
|
||||
more directly misleading half of the bug.
|
||||
- Add a fixture reproducing this shape (a metric with occurrences
|
||||
clustered in a short burst near the start of a much longer ingested
|
||||
window) and a test on `evaluate_promql`/`windowed_rate_series` (or
|
||||
wherever the chosen fix lives) asserting the returned series covers
|
||||
the full requested range, not just the burst.
|
||||
- Out of scope: changing how `KeyPressed`-style occurrence metrics are
|
||||
declared, or adding a dedicated "gauge decay" concept — this is about
|
||||
the query range/x-axis mismatch specifically.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Querying `sum by (code) (rate(KeyPressed[5m]))` (or an equivalent
|
||||
fixture) over a time range much wider than the span of actual
|
||||
occurrences returns a series (or is charted) that visibly reflects
|
||||
rate-zero for the remainder of the range, rather than a line that
|
||||
simply stops.
|
||||
- `PanelChart`'s line chart x-axis reflects the panel's selected/queried
|
||||
time range, not just the min/max timestamp actually present in the
|
||||
returned series — verified by a case where returned data covers only
|
||||
a fraction of the selected range.
|
||||
- Existing `evaluate_promql`/`rate.rs`/`PanelChart` tests for
|
||||
regularly-sampled or continuously-occurring metrics are unaffected.
|
||||
|
||||
## Depends on
|
||||
|
||||
None.
|
||||
Reference in New Issue
Block a user