Metrics rethink: counters as the primitive, histogram()/gauge() derived (ticket 022)

Every metric declaration is now explicitly documented and modeled as
a counter: a match is an occurrence/increment, optionally labeled.
The bucket representation (BucketValue { values: Vec<(Sequence,
Timestamp, MetricPrimitive)> }) already was this; what it lacked was
any way to capture a *value* distinct from the occurrence-count-of-1
marker. Adds RegexToMetricPrimitiveMatcher::role (ParamRole::Label,
the default, or ParamRole::Value — at most one per metric, enforced
at declare time) so a metric can designate one param's captured
number as its per-event value instead of a label.

promql.rs/engine.rs add two new PromQL-evaluable forms built on top
of that value, both consumed as regular PromqlSeriesDto series like
sum()/avg()/rate():
- histogram(metric_name[width]): buckets matching events' values into
  fixed-width buckets, one series per bucket labeled `le`.
- gauge(metric_name): the most-recent value per label combination in
  the queried range.

Existing raw counter queries (metrics with only Label-role params,
sum/avg/rate) are unchanged. Profile files saved before this ticket
still load (role defaults to Label via #[serde(default)]). Metrics.tsx
gains a label/value role picker per param, and the Docs panel explains
the model and the two derived query forms.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
Hugh Kaznowski
2026-09-05 15:18:25 +01:00
co-authored by Claude Sonnet 5
parent 1d0f324c43
commit 0c57ed2953
7 changed files with 388 additions and 33 deletions
+17
View File
@@ -9,6 +9,23 @@ pub struct RegexToMetricPrimitiveMatcher {
pub param_name: String,
pub capture_group: String,
pub kind: MetricPrimitiveKind,
/// Ticket 022: whether this param contributes to the bucket's `labels`
/// (the default) or is the bucket's per-event captured `value` (at most
/// one `Value`-role param per metric — `declare_metric` rejects more
/// than one). `#[serde(default)]` so profile files saved before this
/// field existed still load, with every param defaulting to `Label` —
/// i.e. unchanged behavior (the fixed occurrence-count-of-1 value).
#[serde(default)]
pub role: ParamRole,
}
/// See [`RegexToMetricPrimitiveMatcher::role`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")]
pub enum ParamRole {
#[default]
Label,
Value,
}
/// `rename_all = "lowercase"` matches every other `kind`-style enum exposed
+229 -15
View File
@@ -1,4 +1,4 @@
use crate::data_value::{DataValue, MetricPrimitiveKind, RegexToMetricPrimitiveMatcher};
use crate::data_value::{DataValue, MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher};
use crate::dto::{
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
PromqlSeriesDto, StatsDto, TimelineBucketDto, TimelineParams,
@@ -231,6 +231,13 @@ fn declare_metric(
return Err(errors.join("; "));
}
let value_params = request.params.iter().filter(|matcher| matcher.role == ParamRole::Value).count();
if value_params > 1 {
return Err(format!(
"a metric may declare at most one Value-role param, found {value_params}"
));
}
backfill_metric(index, &request.name, &re, &request.params);
index.insert(
IndexKey::MetricDeclaration { name: request.name },
@@ -263,17 +270,22 @@ fn parse_metric_primitive(raw: &str, kind: MetricPrimitiveKind) -> Option<Metric
}
/// Runs one metric's regex against a single log line and, on a match, appends a
/// bucket value entry.
/// bucket value entry: this is the "counter increment" (ticket 022) — a match is
/// an occurrence at a point in time, optionally labeled.
///
/// Design decision: `Bucket.labels` is the only structured place a `Bucket` key has
/// for a metric's declared params, so v1 folds ALL of a metric's params into
/// `labels` rather than splitting them into "label" vs "value" params. The stored
/// value is therefore a fixed presence marker (`MetricPrimitive::Number(Number::Int(1))`)
/// — a bucket's `values` is the timeseries of *when that label combination occurred*
/// (a count/occurrence series), not a record of the captured values themselves. If a
/// param's capture group doesn't match, or its captured text fails to parse as its
/// declared `kind`, the whole match is skipped for this line rather than storing a
/// partially-labeled bucket.
/// `Bucket.labels` holds every `Label`-role param (the default role — see
/// [`ParamRole`]); at most one param may instead be `Value`-role (enforced at
/// declare time), and its parsed primitive becomes the pushed event's value.
/// With no `Value`-role param, the pushed value is a fixed presence marker
/// (`MetricPrimitive::Number(Number::Int(1))`), matching the original v1
/// behavior where a bucket's `values` was purely the timeseries of *when that
/// label combination occurred* — histogram()/gauge() derived queries
/// (`crate::promql::AggOp`) read this same `value`, so a `Value`-role param is
/// what makes those meaningful (bucketing/reading an occurrence-count-of-1
/// series degenerates to "everything in one bucket"/"always 1"). If a param's
/// capture group doesn't match, or its captured text fails to parse as its
/// declared `kind`, the whole match is skipped for this line rather than
/// storing a partially-labeled bucket.
fn apply_metric_to_log(
index: &mut BTreeMap<IndexKey, DataValue>,
metric_name: &str,
@@ -288,14 +300,22 @@ fn apply_metric_to_log(
};
let mut labels = BTreeMap::new();
let mut value = None;
for matcher in params {
let Some(raw) = captures.name(&matcher.capture_group) else {
return;
};
let Some(value) = parse_metric_primitive(raw.as_str(), matcher.kind) else {
let Some(parsed) = parse_metric_primitive(raw.as_str(), matcher.kind) else {
return;
};
labels.insert(matcher.param_name.clone(), value);
match matcher.role {
ParamRole::Label => {
labels.insert(matcher.param_name.clone(), parsed);
}
ParamRole::Value => {
value = Some(parsed);
}
}
}
let key = IndexKey::Bucket { metric_name: metric_name.to_string(), labels };
@@ -303,7 +323,7 @@ fn apply_metric_to_log(
.entry(key)
.or_insert_with(|| DataValue::BucketValue { values: Vec::new() });
if let DataValue::BucketValue { values } = entry {
values.push((sequence, timestamp, MetricPrimitive::Number(Number::Int(1))));
values.push((sequence, timestamp, value.unwrap_or(MetricPrimitive::Number(Number::Int(1)))));
}
}
@@ -980,6 +1000,51 @@ fn evaluate_promql(
})
.collect()
}
Some(AggOp::Gauge) => {
// Ticket 022: collapse each label combination's counter events
// down to their most-recent value in the queried time range —
// "value at time T" derived from the last matching event, rather
// than a distinct stored gauge kind.
series_points
.into_iter()
.filter_map(|(labels, points)| {
let last = *points.last()?;
Some(PromqlSeriesDto { labels, points: vec![last] })
})
.collect()
}
Some(AggOp::Histogram) => {
// Validated by `promql::parse_query`: `histogram()` never parses
// without one.
let bucket_width =
parsed.bucket_width.expect("histogram() always carries a bucket width");
let last_ts = series_points
.values()
.flat_map(|points| points.iter().map(|(ts, _)| *ts))
.max()
.unwrap_or(0);
// Ticket 022: bucket every matching counter event's value
// (across all label combinations — histogram collapses the
// label dimension the same way sum()/avg() do) into fixed-width
// buckets, one output series per bucket labeled `le` (Prometheus's
// "less-than-or-equal" convention) holding a single
// `(last_ts, count)` point.
let mut counts: BTreeMap<i64, u64> = BTreeMap::new();
for points in series_points.values() {
for (_, value) in points {
let bucket_index = (value / bucket_width).floor() as i64;
*counts.entry(bucket_index).or_insert(0) += 1;
}
}
counts
.into_iter()
.map(|(bucket_index, count)| {
let upper_bound = (bucket_index + 1) as f64 * bucket_width;
let labels = BTreeMap::from([("le".to_string(), upper_bound.to_string())]);
PromqlSeriesDto { labels, points: vec![(last_ts, count as f64)] }
})
.collect()
}
}
}
@@ -1103,16 +1168,32 @@ mod tests {
name: &str,
pattern: &str,
params: Vec<(&str, &str, MetricPrimitiveKind)>,
) -> DeclareMetricRequest {
metric_request_with_roles(
name,
pattern,
params
.into_iter()
.map(|(param_name, capture_group, kind)| (param_name, capture_group, kind, ParamRole::Label))
.collect(),
)
}
fn metric_request_with_roles(
name: &str,
pattern: &str,
params: Vec<(&str, &str, MetricPrimitiveKind, ParamRole)>,
) -> DeclareMetricRequest {
DeclareMetricRequest {
name: name.to_string(),
message_match_regex: pattern.to_string(),
params: params
.into_iter()
.map(|(param_name, capture_group, kind)| RegexToMetricPrimitiveMatcher {
.map(|(param_name, capture_group, kind, role)| RegexToMetricPrimitiveMatcher {
param_name: param_name.to_string(),
capture_group: capture_group.to_string(),
kind,
role,
})
.collect(),
}
@@ -1439,6 +1520,139 @@ mod tests {
assert_eq!(series[0].points, vec![(5, 1.0)]);
}
#[test]
fn declares_metric_with_value_role_param() {
let mut index = BTreeMap::new();
let result = declare_metric(
&mut index,
metric_request_with_roles(
"durations",
r"request path=(?P<path>\S+) duration=(?P<duration>\d+)",
vec![
("path", "path", MetricPrimitiveKind::String, ParamRole::Label),
("duration", "duration", MetricPrimitiveKind::Number, ParamRole::Value),
],
),
);
assert!(result.is_ok());
}
#[test]
fn rejects_metric_with_multiple_value_role_params() {
let mut index = BTreeMap::new();
let result = declare_metric(
&mut index,
metric_request_with_roles(
"bad",
r"a=(?P<a>\d+) b=(?P<b>\d+)",
vec![
("a", "a", MetricPrimitiveKind::Number, ParamRole::Value),
("b", "b", MetricPrimitiveKind::Number, ParamRole::Value),
],
),
);
assert!(result.is_err());
assert!(result.unwrap_err().contains("at most one Value-role param"));
}
#[test]
fn value_role_param_becomes_the_bucket_value_not_a_label() {
let mut index = BTreeMap::new();
declare_metric(
&mut index,
metric_request_with_roles(
"durations",
r"request path=(?P<path>\S+) duration=(?P<duration>\d+)",
vec![
("path", "path", MetricPrimitiveKind::String, ParamRole::Label),
("duration", "duration", MetricPrimitiveKind::Number, ParamRole::Value),
],
),
)
.unwrap();
ingest_log(&mut index, 0, "request path=/a duration=5");
let values: Vec<_> = bucket_values(&index, "durations", None).collect();
assert_eq!(values.len(), 1);
let (labels, (_, _, value)) = &values[0];
assert!(!labels.contains_key("duration"));
assert_eq!(labels.get("path").map(|v| v.to_string()), Some("/a".to_string()));
assert_eq!(*value, MetricPrimitive::Number(Number::Int(5)));
}
fn declare_durations_metric(index: &mut BTreeMap<IndexKey, DataValue>) {
declare_metric(
index,
metric_request_with_roles(
"durations",
r"request path=(?P<path>\S+) duration=(?P<duration>\d+)",
vec![
("path", "path", MetricPrimitiveKind::String, ParamRole::Label),
("duration", "duration", MetricPrimitiveKind::Number, ParamRole::Value),
],
),
)
.unwrap();
}
#[test]
fn gauge_returns_most_recent_value_per_label_combination() {
let mut index = BTreeMap::new();
declare_durations_metric(&mut index);
ingest_log(&mut index, 0, "request path=/a duration=5");
ingest_log(&mut index, 1, "request path=/a duration=9");
ingest_log(&mut index, 2, "request path=/b duration=42");
let parsed = promql::parse_query("gauge(durations)").unwrap();
let mut series = evaluate_promql(&index, &parsed, None, None);
series.sort_by(|a, b| a.labels.get("path").cmp(&b.labels.get("path")));
assert_eq!(series.len(), 2);
assert_eq!(series[0].labels.get("path").map(String::as_str), Some("/a"));
assert_eq!(series[0].points, vec![(1, 9.0)]);
assert_eq!(series[1].labels.get("path").map(String::as_str), Some("/b"));
assert_eq!(series[1].points, vec![(2, 42.0)]);
}
#[test]
fn gauge_respects_time_range_bounds() {
let mut index = BTreeMap::new();
declare_durations_metric(&mut index);
ingest_log(&mut index, 0, "request path=/a duration=5");
ingest_log(&mut index, 5, "request path=/a duration=9");
ingest_log(&mut index, 10, "request path=/a duration=42");
let parsed = promql::parse_query("gauge(durations)").unwrap();
let series = evaluate_promql(&index, &parsed, None, Some(9));
assert_eq!(series.len(), 1);
assert_eq!(series[0].points, vec![(5, 9.0)]);
}
#[test]
fn histogram_buckets_captured_values_by_fixed_width() {
let mut index = BTreeMap::new();
declare_durations_metric(&mut index);
for (seq, duration) in [(0, 2), (1, 4), (2, 12), (3, 15), (4, 21)] {
ingest_log(&mut index, seq, &format!("request path=/a duration={duration}"));
}
let parsed = promql::parse_query("histogram(durations[10])").unwrap();
let mut series = evaluate_promql(&index, &parsed, None, None);
series.sort_by(|a, b| a.labels.get("le").cmp(&b.labels.get("le")));
// durations 2,4 -> bucket [0,10) le=10 (count 2); 12,15 -> [10,20) le=20 (count 2);
// 21 -> [20,30) le=30 (count 1).
assert_eq!(series.len(), 3);
assert_eq!(series[0].labels.get("le").map(String::as_str), Some("10"));
assert_eq!(series[0].points[0].1, 2.0);
assert_eq!(series[1].labels.get("le").map(String::as_str), Some("20"));
assert_eq!(series[1].points[0].1, 2.0);
assert_eq!(series[2].labels.get("le").map(String::as_str), Some("30"));
assert_eq!(series[2].points[0].1, 1.0);
}
#[test]
fn handle_query_promql_rejects_malformed_query() {
let index = BTreeMap::new();
+8 -1
View File
@@ -40,7 +40,7 @@ use std::time::{SystemTime, UNIX_EPOCH};
use serde::{Deserialize, Serialize};
use crate::data_value::{MetricPrimitiveKind, RegexToMetricPrimitiveMatcher};
use crate::data_value::{MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher};
use crate::engine::{DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage};
use crate::settings::Settings;
use tokio::sync::oneshot;
@@ -119,6 +119,10 @@ pub struct MetricParamRecord {
pub param_name: String,
pub capture_group: String,
pub kind: ProfileMetricPrimitiveKind,
/// See `ParamRole`. `#[serde(default)]` so profiles saved before ticket
/// 022 still load, with every param defaulting to `Label`.
#[serde(default)]
pub role: ParamRole,
}
/// A stored metric declaration, matching `DeclareMetricRequest`'s shape (ticket 002).
@@ -141,6 +145,7 @@ impl From<&DeclareMetricRequest> for MetricDeclarationRecord {
param_name: matcher.param_name.clone(),
capture_group: matcher.capture_group.clone(),
kind: matcher.kind.into(),
role: matcher.role,
})
.collect(),
}
@@ -159,6 +164,7 @@ impl From<&MetricDeclarationRecord> for DeclareMetricRequest {
param_name: param.param_name.clone(),
capture_group: param.capture_group.clone(),
kind: param.kind.into(),
role: param.role,
})
.collect(),
}
@@ -708,6 +714,7 @@ mod tests {
param_name: param_name.to_string(),
capture_group: capture_group.to_string(),
kind: MetricPrimitiveKind::Number,
role: ParamRole::Label,
}],
}
}
+91 -13
View File
@@ -14,21 +14,34 @@
//! ```text
//! query := aggregation | vector_selector
//! aggregation := agg_op "(" vector_selector ")"
//! agg_op := "sum" | "avg" | "rate"
//! agg_op := "sum" | "avg" | "rate" | "histogram" | "gauge"
//! vector_selector := metric_name label_matchers? range_selector?
//! metric_name := [a-zA-Z_][a-zA-Z0-9_]*
//! label_matchers := "{" label_matcher ("," label_matcher)* "}"
//! label_matcher := label_name "=" quoted_string
//! label_name := [a-zA-Z_][a-zA-Z0-9_]*
//! quoted_string := '"' (any char except '"' or '\')* '"'
//! range_selector := "[" digits duration_unit "]"
//! range_selector := "[" digits duration_unit? "]"
//! duration_unit := "s" | "m" | "h" | "d"
//! ```
//!
//! Only exact-match label matchers (`=`) are supported — PromQL's `!=`, `=~`, `!~`
//! are out of scope for this ticket's required subset. `rate()` requires its inner
//! vector selector to carry a range selector (matching real PromQL), and is rejected
//! at parse time otherwise.
//! vector selector to carry a range selector with a duration unit (matching real
//! PromQL), and is rejected at parse time otherwise.
//!
//! Ticket 022 (metrics-as-counters rethink) adds two derived forms on top of the
//! same grammar, both consumed as regular `PromqlSeriesDto` series like every
//! other query shape:
//! - `histogram(metric_name[width])`: buckets every matching counter event's
//! (coerced-to-f64) value into fixed-width buckets of size `width` (a bare
//! number, no duration unit — reusing the `[...]` range-selector slot for a
//! different purpose than `rate()`'s duration), and emits one series per
//! bucket, labeled `le` (Prometheus's "less-than-or-equal" convention) with
//! that bucket's upper bound, holding a single `(timestamp_ms, count)` point.
//! - `gauge(metric_name)`: takes no range selector; emits the most-recent
//! matching counter event's value per label combination as a single point,
//! i.e. the counter's series collapsed down to its last value.
use std::time::Duration;
@@ -37,6 +50,11 @@ pub enum AggOp {
Sum,
Avg,
Rate,
/// Ticket 022: buckets counter values into fixed-width buckets. Requires
/// `bucket_width`.
Histogram,
/// Ticket 022: the most-recent counter value per label combination.
Gauge,
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -50,7 +68,18 @@ pub struct PromqlQuery {
pub agg: Option<AggOp>,
pub metric_name: String,
pub matchers: Vec<LabelMatcher>,
/// Duration range selector, e.g. `[5m]`. Only meaningful for `rate()`.
pub range: Option<Duration>,
/// Bare-number range selector, e.g. `[10]`. Only meaningful for
/// `histogram()`, where it's the bucket width.
pub bucket_width: Option<f64>,
}
/// What a `[...]` range selector turned out to hold, disambiguated by whether
/// a duration unit followed the digits.
enum RangeSelector {
Duration(Duration),
BareWidth(f64),
}
/// Parses a PromQL query string per the grammar documented above. Returns a
@@ -66,6 +95,11 @@ pub fn parse_query(input: &str) -> Result<PromqlQuery, String> {
if query.agg == Some(AggOp::Rate) && query.range.is_none() {
return Err("rate() requires a range selector, e.g. rate(my_metric[5m])".to_string());
}
if query.agg == Some(AggOp::Histogram) && query.bucket_width.is_none() {
return Err(
"histogram() requires a bucket width, e.g. histogram(my_metric[10])".to_string(),
);
}
Ok(query)
}
@@ -180,8 +214,10 @@ impl<'a> Parser<'a> {
Ok(matchers)
}
/// `[<digits><unit>]`, e.g. `[5m]`.
fn parse_range_selector(&mut self) -> Result<Option<Duration>, String> {
/// `[<digits><unit>]` (e.g. `[5m]`, a duration — used by `rate()`) or a
/// bare `[<digits>]` with no unit (a plain number — used by `histogram()`
/// as a bucket width).
fn parse_range_selector(&mut self) -> Result<Option<RangeSelector>, String> {
self.skip_whitespace();
if self.peek() != Some('[') {
return Ok(None);
@@ -190,16 +226,21 @@ impl<'a> Parser<'a> {
let rest = self.remaining();
let digits_end = rest
.find(|c: char| !c.is_ascii_digit())
.ok_or("unterminated range selector (missing unit and ']')")?;
.ok_or("unterminated range selector (missing ']')")?;
if digits_end == 0 {
return Err("range selector must start with a number, e.g. [5m]".to_string());
return Err("range selector must start with a number, e.g. [5m] or [10]".to_string());
}
let digits: i64 = rest[..digits_end]
.parse()
.map_err(|_| "range selector duration is not a valid number".to_string())?;
.map_err(|_| "range selector number is not a valid number".to_string())?;
self.pos += digits_end;
let unit = self.peek().ok_or("range selector is missing a duration unit")?;
if self.peek() == Some(']') {
self.expect_char(']')?;
return Ok(Some(RangeSelector::BareWidth(digits as f64)));
}
let unit = self.peek().ok_or("range selector is missing a duration unit or ']'")?;
let seconds_per_unit: i64 = match unit {
's' => 1,
'm' => 60,
@@ -213,14 +254,20 @@ impl<'a> Parser<'a> {
};
self.pos += unit.len_utf8();
self.expect_char(']')?;
Ok(Some(Duration::from_secs((digits * seconds_per_unit).max(0) as u64)))
Ok(Some(RangeSelector::Duration(Duration::from_secs(
(digits * seconds_per_unit).max(0) as u64,
))))
}
fn parse_vector_selector(&mut self) -> Result<PromqlQuery, String> {
let metric_name = self.parse_identifier()?;
let matchers = self.parse_label_matchers()?;
let range = self.parse_range_selector()?;
Ok(PromqlQuery { agg: None, metric_name, matchers, range })
let (range, bucket_width) = match self.parse_range_selector()? {
None => (None, None),
Some(RangeSelector::Duration(d)) => (Some(d), None),
Some(RangeSelector::BareWidth(w)) => (None, Some(w)),
};
Ok(PromqlQuery { agg: None, metric_name, matchers, range, bucket_width })
}
fn parse_query(&mut self) -> Result<PromqlQuery, String> {
@@ -231,6 +278,8 @@ impl<'a> Parser<'a> {
"sum" => Some(AggOp::Sum),
"avg" => Some(AggOp::Avg),
"rate" => Some(AggOp::Rate),
"histogram" => Some(AggOp::Histogram),
"gauge" => Some(AggOp::Gauge),
_ => None,
};
self.skip_whitespace();
@@ -366,4 +415,33 @@ mod tests {
let err = parse_query("requests extra").unwrap_err();
assert!(err.contains("unexpected trailing input"));
}
#[test]
fn parses_histogram_with_bare_width() {
let query = parse_query("histogram(latency_ms[10])").unwrap();
assert_eq!(query.agg, Some(AggOp::Histogram));
assert_eq!(query.metric_name, "latency_ms");
assert_eq!(query.bucket_width, Some(10.0));
assert_eq!(query.range, None);
}
#[test]
fn rejects_histogram_without_width() {
let err = parse_query("histogram(latency_ms)").unwrap_err();
assert!(err.contains("bucket width"));
}
#[test]
fn parses_gauge_with_no_range_selector() {
let query = parse_query(r#"gauge(connections{host="a"})"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Gauge));
assert_eq!(query.matchers, vec![LabelMatcher { name: "host".to_string(), value: "a".to_string() }]);
}
#[test]
fn rate_range_selector_is_still_a_duration_not_a_bare_width() {
let query = parse_query("rate(requests[5m])").unwrap();
assert_eq!(query.range, Some(Duration::from_secs(300)));
assert_eq!(query.bucket_width, None);
}
}