Metrics rethink: counters as the primitive, histogram()/gauge() derived (ticket 022)
Every metric declaration is now explicitly documented and modeled as
a counter: a match is an occurrence/increment, optionally labeled.
The bucket representation (BucketValue { values: Vec<(Sequence,
Timestamp, MetricPrimitive)> }) already was this; what it lacked was
any way to capture a *value* distinct from the occurrence-count-of-1
marker. Adds RegexToMetricPrimitiveMatcher::role (ParamRole::Label,
the default, or ParamRole::Value — at most one per metric, enforced
at declare time) so a metric can designate one param's captured
number as its per-event value instead of a label.
promql.rs/engine.rs add two new PromQL-evaluable forms built on top
of that value, both consumed as regular PromqlSeriesDto series like
sum()/avg()/rate():
- histogram(metric_name[width]): buckets matching events' values into
fixed-width buckets, one series per bucket labeled `le`.
- gauge(metric_name): the most-recent value per label combination in
the queried range.
Existing raw counter queries (metrics with only Label-role params,
sum/avg/rate) are unchanged. Profile files saved before this ticket
still load (role defaults to Label via #[serde(default)]). Metrics.tsx
gains a label/value role picker per param, and the Docs panel explains
the model and the two derived query forms.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
1d0f324c43
commit
0c57ed2953
@@ -9,6 +9,23 @@ pub struct RegexToMetricPrimitiveMatcher {
|
|||||||
pub param_name: String,
|
pub param_name: String,
|
||||||
pub capture_group: String,
|
pub capture_group: String,
|
||||||
pub kind: MetricPrimitiveKind,
|
pub kind: MetricPrimitiveKind,
|
||||||
|
/// Ticket 022: whether this param contributes to the bucket's `labels`
|
||||||
|
/// (the default) or is the bucket's per-event captured `value` (at most
|
||||||
|
/// one `Value`-role param per metric — `declare_metric` rejects more
|
||||||
|
/// than one). `#[serde(default)]` so profile files saved before this
|
||||||
|
/// field existed still load, with every param defaulting to `Label` —
|
||||||
|
/// i.e. unchanged behavior (the fixed occurrence-count-of-1 value).
|
||||||
|
#[serde(default)]
|
||||||
|
pub role: ParamRole,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// See [`RegexToMetricPrimitiveMatcher::role`].
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||||
|
#[serde(rename_all = "lowercase")]
|
||||||
|
pub enum ParamRole {
|
||||||
|
#[default]
|
||||||
|
Label,
|
||||||
|
Value,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `rename_all = "lowercase"` matches every other `kind`-style enum exposed
|
/// `rename_all = "lowercase"` matches every other `kind`-style enum exposed
|
||||||
|
|||||||
+229
-15
@@ -1,4 +1,4 @@
|
|||||||
use crate::data_value::{DataValue, MetricPrimitiveKind, RegexToMetricPrimitiveMatcher};
|
use crate::data_value::{DataValue, MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher};
|
||||||
use crate::dto::{
|
use crate::dto::{
|
||||||
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
|
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
|
||||||
PromqlSeriesDto, StatsDto, TimelineBucketDto, TimelineParams,
|
PromqlSeriesDto, StatsDto, TimelineBucketDto, TimelineParams,
|
||||||
@@ -231,6 +231,13 @@ fn declare_metric(
|
|||||||
return Err(errors.join("; "));
|
return Err(errors.join("; "));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
let value_params = request.params.iter().filter(|matcher| matcher.role == ParamRole::Value).count();
|
||||||
|
if value_params > 1 {
|
||||||
|
return Err(format!(
|
||||||
|
"a metric may declare at most one Value-role param, found {value_params}"
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
backfill_metric(index, &request.name, &re, &request.params);
|
backfill_metric(index, &request.name, &re, &request.params);
|
||||||
index.insert(
|
index.insert(
|
||||||
IndexKey::MetricDeclaration { name: request.name },
|
IndexKey::MetricDeclaration { name: request.name },
|
||||||
@@ -263,17 +270,22 @@ fn parse_metric_primitive(raw: &str, kind: MetricPrimitiveKind) -> Option<Metric
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Runs one metric's regex against a single log line and, on a match, appends a
|
/// Runs one metric's regex against a single log line and, on a match, appends a
|
||||||
/// bucket value entry.
|
/// bucket value entry: this is the "counter increment" (ticket 022) — a match is
|
||||||
|
/// an occurrence at a point in time, optionally labeled.
|
||||||
///
|
///
|
||||||
/// Design decision: `Bucket.labels` is the only structured place a `Bucket` key has
|
/// `Bucket.labels` holds every `Label`-role param (the default role — see
|
||||||
/// for a metric's declared params, so v1 folds ALL of a metric's params into
|
/// [`ParamRole`]); at most one param may instead be `Value`-role (enforced at
|
||||||
/// `labels` rather than splitting them into "label" vs "value" params. The stored
|
/// declare time), and its parsed primitive becomes the pushed event's value.
|
||||||
/// value is therefore a fixed presence marker (`MetricPrimitive::Number(Number::Int(1))`)
|
/// With no `Value`-role param, the pushed value is a fixed presence marker
|
||||||
/// — a bucket's `values` is the timeseries of *when that label combination occurred*
|
/// (`MetricPrimitive::Number(Number::Int(1))`), matching the original v1
|
||||||
/// (a count/occurrence series), not a record of the captured values themselves. If a
|
/// behavior where a bucket's `values` was purely the timeseries of *when that
|
||||||
/// param's capture group doesn't match, or its captured text fails to parse as its
|
/// label combination occurred* — histogram()/gauge() derived queries
|
||||||
/// declared `kind`, the whole match is skipped for this line rather than storing a
|
/// (`crate::promql::AggOp`) read this same `value`, so a `Value`-role param is
|
||||||
/// partially-labeled bucket.
|
/// what makes those meaningful (bucketing/reading an occurrence-count-of-1
|
||||||
|
/// series degenerates to "everything in one bucket"/"always 1"). If a param's
|
||||||
|
/// capture group doesn't match, or its captured text fails to parse as its
|
||||||
|
/// declared `kind`, the whole match is skipped for this line rather than
|
||||||
|
/// storing a partially-labeled bucket.
|
||||||
fn apply_metric_to_log(
|
fn apply_metric_to_log(
|
||||||
index: &mut BTreeMap<IndexKey, DataValue>,
|
index: &mut BTreeMap<IndexKey, DataValue>,
|
||||||
metric_name: &str,
|
metric_name: &str,
|
||||||
@@ -288,14 +300,22 @@ fn apply_metric_to_log(
|
|||||||
};
|
};
|
||||||
|
|
||||||
let mut labels = BTreeMap::new();
|
let mut labels = BTreeMap::new();
|
||||||
|
let mut value = None;
|
||||||
for matcher in params {
|
for matcher in params {
|
||||||
let Some(raw) = captures.name(&matcher.capture_group) else {
|
let Some(raw) = captures.name(&matcher.capture_group) else {
|
||||||
return;
|
return;
|
||||||
};
|
};
|
||||||
let Some(value) = parse_metric_primitive(raw.as_str(), matcher.kind) else {
|
let Some(parsed) = parse_metric_primitive(raw.as_str(), matcher.kind) else {
|
||||||
return;
|
return;
|
||||||
};
|
};
|
||||||
labels.insert(matcher.param_name.clone(), value);
|
match matcher.role {
|
||||||
|
ParamRole::Label => {
|
||||||
|
labels.insert(matcher.param_name.clone(), parsed);
|
||||||
|
}
|
||||||
|
ParamRole::Value => {
|
||||||
|
value = Some(parsed);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let key = IndexKey::Bucket { metric_name: metric_name.to_string(), labels };
|
let key = IndexKey::Bucket { metric_name: metric_name.to_string(), labels };
|
||||||
@@ -303,7 +323,7 @@ fn apply_metric_to_log(
|
|||||||
.entry(key)
|
.entry(key)
|
||||||
.or_insert_with(|| DataValue::BucketValue { values: Vec::new() });
|
.or_insert_with(|| DataValue::BucketValue { values: Vec::new() });
|
||||||
if let DataValue::BucketValue { values } = entry {
|
if let DataValue::BucketValue { values } = entry {
|
||||||
values.push((sequence, timestamp, MetricPrimitive::Number(Number::Int(1))));
|
values.push((sequence, timestamp, value.unwrap_or(MetricPrimitive::Number(Number::Int(1)))));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -980,6 +1000,51 @@ fn evaluate_promql(
|
|||||||
})
|
})
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
Some(AggOp::Gauge) => {
|
||||||
|
// Ticket 022: collapse each label combination's counter events
|
||||||
|
// down to their most-recent value in the queried time range —
|
||||||
|
// "value at time T" derived from the last matching event, rather
|
||||||
|
// than a distinct stored gauge kind.
|
||||||
|
series_points
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|(labels, points)| {
|
||||||
|
let last = *points.last()?;
|
||||||
|
Some(PromqlSeriesDto { labels, points: vec![last] })
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
Some(AggOp::Histogram) => {
|
||||||
|
// Validated by `promql::parse_query`: `histogram()` never parses
|
||||||
|
// without one.
|
||||||
|
let bucket_width =
|
||||||
|
parsed.bucket_width.expect("histogram() always carries a bucket width");
|
||||||
|
let last_ts = series_points
|
||||||
|
.values()
|
||||||
|
.flat_map(|points| points.iter().map(|(ts, _)| *ts))
|
||||||
|
.max()
|
||||||
|
.unwrap_or(0);
|
||||||
|
// Ticket 022: bucket every matching counter event's value
|
||||||
|
// (across all label combinations — histogram collapses the
|
||||||
|
// label dimension the same way sum()/avg() do) into fixed-width
|
||||||
|
// buckets, one output series per bucket labeled `le` (Prometheus's
|
||||||
|
// "less-than-or-equal" convention) holding a single
|
||||||
|
// `(last_ts, count)` point.
|
||||||
|
let mut counts: BTreeMap<i64, u64> = BTreeMap::new();
|
||||||
|
for points in series_points.values() {
|
||||||
|
for (_, value) in points {
|
||||||
|
let bucket_index = (value / bucket_width).floor() as i64;
|
||||||
|
*counts.entry(bucket_index).or_insert(0) += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
counts
|
||||||
|
.into_iter()
|
||||||
|
.map(|(bucket_index, count)| {
|
||||||
|
let upper_bound = (bucket_index + 1) as f64 * bucket_width;
|
||||||
|
let labels = BTreeMap::from([("le".to_string(), upper_bound.to_string())]);
|
||||||
|
PromqlSeriesDto { labels, points: vec![(last_ts, count as f64)] }
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1103,16 +1168,32 @@ mod tests {
|
|||||||
name: &str,
|
name: &str,
|
||||||
pattern: &str,
|
pattern: &str,
|
||||||
params: Vec<(&str, &str, MetricPrimitiveKind)>,
|
params: Vec<(&str, &str, MetricPrimitiveKind)>,
|
||||||
|
) -> DeclareMetricRequest {
|
||||||
|
metric_request_with_roles(
|
||||||
|
name,
|
||||||
|
pattern,
|
||||||
|
params
|
||||||
|
.into_iter()
|
||||||
|
.map(|(param_name, capture_group, kind)| (param_name, capture_group, kind, ParamRole::Label))
|
||||||
|
.collect(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn metric_request_with_roles(
|
||||||
|
name: &str,
|
||||||
|
pattern: &str,
|
||||||
|
params: Vec<(&str, &str, MetricPrimitiveKind, ParamRole)>,
|
||||||
) -> DeclareMetricRequest {
|
) -> DeclareMetricRequest {
|
||||||
DeclareMetricRequest {
|
DeclareMetricRequest {
|
||||||
name: name.to_string(),
|
name: name.to_string(),
|
||||||
message_match_regex: pattern.to_string(),
|
message_match_regex: pattern.to_string(),
|
||||||
params: params
|
params: params
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|(param_name, capture_group, kind)| RegexToMetricPrimitiveMatcher {
|
.map(|(param_name, capture_group, kind, role)| RegexToMetricPrimitiveMatcher {
|
||||||
param_name: param_name.to_string(),
|
param_name: param_name.to_string(),
|
||||||
capture_group: capture_group.to_string(),
|
capture_group: capture_group.to_string(),
|
||||||
kind,
|
kind,
|
||||||
|
role,
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
}
|
}
|
||||||
@@ -1439,6 +1520,139 @@ mod tests {
|
|||||||
assert_eq!(series[0].points, vec![(5, 1.0)]);
|
assert_eq!(series[0].points, vec![(5, 1.0)]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn declares_metric_with_value_role_param() {
|
||||||
|
let mut index = BTreeMap::new();
|
||||||
|
let result = declare_metric(
|
||||||
|
&mut index,
|
||||||
|
metric_request_with_roles(
|
||||||
|
"durations",
|
||||||
|
r"request path=(?P<path>\S+) duration=(?P<duration>\d+)",
|
||||||
|
vec![
|
||||||
|
("path", "path", MetricPrimitiveKind::String, ParamRole::Label),
|
||||||
|
("duration", "duration", MetricPrimitiveKind::Number, ParamRole::Value),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert!(result.is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_metric_with_multiple_value_role_params() {
|
||||||
|
let mut index = BTreeMap::new();
|
||||||
|
let result = declare_metric(
|
||||||
|
&mut index,
|
||||||
|
metric_request_with_roles(
|
||||||
|
"bad",
|
||||||
|
r"a=(?P<a>\d+) b=(?P<b>\d+)",
|
||||||
|
vec![
|
||||||
|
("a", "a", MetricPrimitiveKind::Number, ParamRole::Value),
|
||||||
|
("b", "b", MetricPrimitiveKind::Number, ParamRole::Value),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert!(result.is_err());
|
||||||
|
assert!(result.unwrap_err().contains("at most one Value-role param"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn value_role_param_becomes_the_bucket_value_not_a_label() {
|
||||||
|
let mut index = BTreeMap::new();
|
||||||
|
declare_metric(
|
||||||
|
&mut index,
|
||||||
|
metric_request_with_roles(
|
||||||
|
"durations",
|
||||||
|
r"request path=(?P<path>\S+) duration=(?P<duration>\d+)",
|
||||||
|
vec![
|
||||||
|
("path", "path", MetricPrimitiveKind::String, ParamRole::Label),
|
||||||
|
("duration", "duration", MetricPrimitiveKind::Number, ParamRole::Value),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
ingest_log(&mut index, 0, "request path=/a duration=5");
|
||||||
|
|
||||||
|
let values: Vec<_> = bucket_values(&index, "durations", None).collect();
|
||||||
|
assert_eq!(values.len(), 1);
|
||||||
|
let (labels, (_, _, value)) = &values[0];
|
||||||
|
assert!(!labels.contains_key("duration"));
|
||||||
|
assert_eq!(labels.get("path").map(|v| v.to_string()), Some("/a".to_string()));
|
||||||
|
assert_eq!(*value, MetricPrimitive::Number(Number::Int(5)));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn declare_durations_metric(index: &mut BTreeMap<IndexKey, DataValue>) {
|
||||||
|
declare_metric(
|
||||||
|
index,
|
||||||
|
metric_request_with_roles(
|
||||||
|
"durations",
|
||||||
|
r"request path=(?P<path>\S+) duration=(?P<duration>\d+)",
|
||||||
|
vec![
|
||||||
|
("path", "path", MetricPrimitiveKind::String, ParamRole::Label),
|
||||||
|
("duration", "duration", MetricPrimitiveKind::Number, ParamRole::Value),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauge_returns_most_recent_value_per_label_combination() {
|
||||||
|
let mut index = BTreeMap::new();
|
||||||
|
declare_durations_metric(&mut index);
|
||||||
|
ingest_log(&mut index, 0, "request path=/a duration=5");
|
||||||
|
ingest_log(&mut index, 1, "request path=/a duration=9");
|
||||||
|
ingest_log(&mut index, 2, "request path=/b duration=42");
|
||||||
|
|
||||||
|
let parsed = promql::parse_query("gauge(durations)").unwrap();
|
||||||
|
let mut series = evaluate_promql(&index, &parsed, None, None);
|
||||||
|
series.sort_by(|a, b| a.labels.get("path").cmp(&b.labels.get("path")));
|
||||||
|
|
||||||
|
assert_eq!(series.len(), 2);
|
||||||
|
assert_eq!(series[0].labels.get("path").map(String::as_str), Some("/a"));
|
||||||
|
assert_eq!(series[0].points, vec![(1, 9.0)]);
|
||||||
|
assert_eq!(series[1].labels.get("path").map(String::as_str), Some("/b"));
|
||||||
|
assert_eq!(series[1].points, vec![(2, 42.0)]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauge_respects_time_range_bounds() {
|
||||||
|
let mut index = BTreeMap::new();
|
||||||
|
declare_durations_metric(&mut index);
|
||||||
|
ingest_log(&mut index, 0, "request path=/a duration=5");
|
||||||
|
ingest_log(&mut index, 5, "request path=/a duration=9");
|
||||||
|
ingest_log(&mut index, 10, "request path=/a duration=42");
|
||||||
|
|
||||||
|
let parsed = promql::parse_query("gauge(durations)").unwrap();
|
||||||
|
let series = evaluate_promql(&index, &parsed, None, Some(9));
|
||||||
|
|
||||||
|
assert_eq!(series.len(), 1);
|
||||||
|
assert_eq!(series[0].points, vec![(5, 9.0)]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn histogram_buckets_captured_values_by_fixed_width() {
|
||||||
|
let mut index = BTreeMap::new();
|
||||||
|
declare_durations_metric(&mut index);
|
||||||
|
for (seq, duration) in [(0, 2), (1, 4), (2, 12), (3, 15), (4, 21)] {
|
||||||
|
ingest_log(&mut index, seq, &format!("request path=/a duration={duration}"));
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed = promql::parse_query("histogram(durations[10])").unwrap();
|
||||||
|
let mut series = evaluate_promql(&index, &parsed, None, None);
|
||||||
|
series.sort_by(|a, b| a.labels.get("le").cmp(&b.labels.get("le")));
|
||||||
|
|
||||||
|
// durations 2,4 -> bucket [0,10) le=10 (count 2); 12,15 -> [10,20) le=20 (count 2);
|
||||||
|
// 21 -> [20,30) le=30 (count 1).
|
||||||
|
assert_eq!(series.len(), 3);
|
||||||
|
assert_eq!(series[0].labels.get("le").map(String::as_str), Some("10"));
|
||||||
|
assert_eq!(series[0].points[0].1, 2.0);
|
||||||
|
assert_eq!(series[1].labels.get("le").map(String::as_str), Some("20"));
|
||||||
|
assert_eq!(series[1].points[0].1, 2.0);
|
||||||
|
assert_eq!(series[2].labels.get("le").map(String::as_str), Some("30"));
|
||||||
|
assert_eq!(series[2].points[0].1, 1.0);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn handle_query_promql_rejects_malformed_query() {
|
fn handle_query_promql_rejects_malformed_query() {
|
||||||
let index = BTreeMap::new();
|
let index = BTreeMap::new();
|
||||||
|
|||||||
+8
-1
@@ -40,7 +40,7 @@ use std::time::{SystemTime, UNIX_EPOCH};
|
|||||||
|
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
use crate::data_value::{MetricPrimitiveKind, RegexToMetricPrimitiveMatcher};
|
use crate::data_value::{MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher};
|
||||||
use crate::engine::{DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage};
|
use crate::engine::{DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage};
|
||||||
use crate::settings::Settings;
|
use crate::settings::Settings;
|
||||||
use tokio::sync::oneshot;
|
use tokio::sync::oneshot;
|
||||||
@@ -119,6 +119,10 @@ pub struct MetricParamRecord {
|
|||||||
pub param_name: String,
|
pub param_name: String,
|
||||||
pub capture_group: String,
|
pub capture_group: String,
|
||||||
pub kind: ProfileMetricPrimitiveKind,
|
pub kind: ProfileMetricPrimitiveKind,
|
||||||
|
/// See `ParamRole`. `#[serde(default)]` so profiles saved before ticket
|
||||||
|
/// 022 still load, with every param defaulting to `Label`.
|
||||||
|
#[serde(default)]
|
||||||
|
pub role: ParamRole,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A stored metric declaration, matching `DeclareMetricRequest`'s shape (ticket 002).
|
/// A stored metric declaration, matching `DeclareMetricRequest`'s shape (ticket 002).
|
||||||
@@ -141,6 +145,7 @@ impl From<&DeclareMetricRequest> for MetricDeclarationRecord {
|
|||||||
param_name: matcher.param_name.clone(),
|
param_name: matcher.param_name.clone(),
|
||||||
capture_group: matcher.capture_group.clone(),
|
capture_group: matcher.capture_group.clone(),
|
||||||
kind: matcher.kind.into(),
|
kind: matcher.kind.into(),
|
||||||
|
role: matcher.role,
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
}
|
}
|
||||||
@@ -159,6 +164,7 @@ impl From<&MetricDeclarationRecord> for DeclareMetricRequest {
|
|||||||
param_name: param.param_name.clone(),
|
param_name: param.param_name.clone(),
|
||||||
capture_group: param.capture_group.clone(),
|
capture_group: param.capture_group.clone(),
|
||||||
kind: param.kind.into(),
|
kind: param.kind.into(),
|
||||||
|
role: param.role,
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
}
|
}
|
||||||
@@ -708,6 +714,7 @@ mod tests {
|
|||||||
param_name: param_name.to_string(),
|
param_name: param_name.to_string(),
|
||||||
capture_group: capture_group.to_string(),
|
capture_group: capture_group.to_string(),
|
||||||
kind: MetricPrimitiveKind::Number,
|
kind: MetricPrimitiveKind::Number,
|
||||||
|
role: ParamRole::Label,
|
||||||
}],
|
}],
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+91
-13
@@ -14,21 +14,34 @@
|
|||||||
//! ```text
|
//! ```text
|
||||||
//! query := aggregation | vector_selector
|
//! query := aggregation | vector_selector
|
||||||
//! aggregation := agg_op "(" vector_selector ")"
|
//! aggregation := agg_op "(" vector_selector ")"
|
||||||
//! agg_op := "sum" | "avg" | "rate"
|
//! agg_op := "sum" | "avg" | "rate" | "histogram" | "gauge"
|
||||||
//! vector_selector := metric_name label_matchers? range_selector?
|
//! vector_selector := metric_name label_matchers? range_selector?
|
||||||
//! metric_name := [a-zA-Z_][a-zA-Z0-9_]*
|
//! metric_name := [a-zA-Z_][a-zA-Z0-9_]*
|
||||||
//! label_matchers := "{" label_matcher ("," label_matcher)* "}"
|
//! label_matchers := "{" label_matcher ("," label_matcher)* "}"
|
||||||
//! label_matcher := label_name "=" quoted_string
|
//! label_matcher := label_name "=" quoted_string
|
||||||
//! label_name := [a-zA-Z_][a-zA-Z0-9_]*
|
//! label_name := [a-zA-Z_][a-zA-Z0-9_]*
|
||||||
//! quoted_string := '"' (any char except '"' or '\')* '"'
|
//! quoted_string := '"' (any char except '"' or '\')* '"'
|
||||||
//! range_selector := "[" digits duration_unit "]"
|
//! range_selector := "[" digits duration_unit? "]"
|
||||||
//! duration_unit := "s" | "m" | "h" | "d"
|
//! duration_unit := "s" | "m" | "h" | "d"
|
||||||
//! ```
|
//! ```
|
||||||
//!
|
//!
|
||||||
//! Only exact-match label matchers (`=`) are supported — PromQL's `!=`, `=~`, `!~`
|
//! Only exact-match label matchers (`=`) are supported — PromQL's `!=`, `=~`, `!~`
|
||||||
//! are out of scope for this ticket's required subset. `rate()` requires its inner
|
//! are out of scope for this ticket's required subset. `rate()` requires its inner
|
||||||
//! vector selector to carry a range selector (matching real PromQL), and is rejected
|
//! vector selector to carry a range selector with a duration unit (matching real
|
||||||
//! at parse time otherwise.
|
//! PromQL), and is rejected at parse time otherwise.
|
||||||
|
//!
|
||||||
|
//! Ticket 022 (metrics-as-counters rethink) adds two derived forms on top of the
|
||||||
|
//! same grammar, both consumed as regular `PromqlSeriesDto` series like every
|
||||||
|
//! other query shape:
|
||||||
|
//! - `histogram(metric_name[width])`: buckets every matching counter event's
|
||||||
|
//! (coerced-to-f64) value into fixed-width buckets of size `width` (a bare
|
||||||
|
//! number, no duration unit — reusing the `[...]` range-selector slot for a
|
||||||
|
//! different purpose than `rate()`'s duration), and emits one series per
|
||||||
|
//! bucket, labeled `le` (Prometheus's "less-than-or-equal" convention) with
|
||||||
|
//! that bucket's upper bound, holding a single `(timestamp_ms, count)` point.
|
||||||
|
//! - `gauge(metric_name)`: takes no range selector; emits the most-recent
|
||||||
|
//! matching counter event's value per label combination as a single point,
|
||||||
|
//! i.e. the counter's series collapsed down to its last value.
|
||||||
|
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
@@ -37,6 +50,11 @@ pub enum AggOp {
|
|||||||
Sum,
|
Sum,
|
||||||
Avg,
|
Avg,
|
||||||
Rate,
|
Rate,
|
||||||
|
/// Ticket 022: buckets counter values into fixed-width buckets. Requires
|
||||||
|
/// `bucket_width`.
|
||||||
|
Histogram,
|
||||||
|
/// Ticket 022: the most-recent counter value per label combination.
|
||||||
|
Gauge,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
@@ -50,7 +68,18 @@ pub struct PromqlQuery {
|
|||||||
pub agg: Option<AggOp>,
|
pub agg: Option<AggOp>,
|
||||||
pub metric_name: String,
|
pub metric_name: String,
|
||||||
pub matchers: Vec<LabelMatcher>,
|
pub matchers: Vec<LabelMatcher>,
|
||||||
|
/// Duration range selector, e.g. `[5m]`. Only meaningful for `rate()`.
|
||||||
pub range: Option<Duration>,
|
pub range: Option<Duration>,
|
||||||
|
/// Bare-number range selector, e.g. `[10]`. Only meaningful for
|
||||||
|
/// `histogram()`, where it's the bucket width.
|
||||||
|
pub bucket_width: Option<f64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What a `[...]` range selector turned out to hold, disambiguated by whether
|
||||||
|
/// a duration unit followed the digits.
|
||||||
|
enum RangeSelector {
|
||||||
|
Duration(Duration),
|
||||||
|
BareWidth(f64),
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Parses a PromQL query string per the grammar documented above. Returns a
|
/// Parses a PromQL query string per the grammar documented above. Returns a
|
||||||
@@ -66,6 +95,11 @@ pub fn parse_query(input: &str) -> Result<PromqlQuery, String> {
|
|||||||
if query.agg == Some(AggOp::Rate) && query.range.is_none() {
|
if query.agg == Some(AggOp::Rate) && query.range.is_none() {
|
||||||
return Err("rate() requires a range selector, e.g. rate(my_metric[5m])".to_string());
|
return Err("rate() requires a range selector, e.g. rate(my_metric[5m])".to_string());
|
||||||
}
|
}
|
||||||
|
if query.agg == Some(AggOp::Histogram) && query.bucket_width.is_none() {
|
||||||
|
return Err(
|
||||||
|
"histogram() requires a bucket width, e.g. histogram(my_metric[10])".to_string(),
|
||||||
|
);
|
||||||
|
}
|
||||||
Ok(query)
|
Ok(query)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -180,8 +214,10 @@ impl<'a> Parser<'a> {
|
|||||||
Ok(matchers)
|
Ok(matchers)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `[<digits><unit>]`, e.g. `[5m]`.
|
/// `[<digits><unit>]` (e.g. `[5m]`, a duration — used by `rate()`) or a
|
||||||
fn parse_range_selector(&mut self) -> Result<Option<Duration>, String> {
|
/// bare `[<digits>]` with no unit (a plain number — used by `histogram()`
|
||||||
|
/// as a bucket width).
|
||||||
|
fn parse_range_selector(&mut self) -> Result<Option<RangeSelector>, String> {
|
||||||
self.skip_whitespace();
|
self.skip_whitespace();
|
||||||
if self.peek() != Some('[') {
|
if self.peek() != Some('[') {
|
||||||
return Ok(None);
|
return Ok(None);
|
||||||
@@ -190,16 +226,21 @@ impl<'a> Parser<'a> {
|
|||||||
let rest = self.remaining();
|
let rest = self.remaining();
|
||||||
let digits_end = rest
|
let digits_end = rest
|
||||||
.find(|c: char| !c.is_ascii_digit())
|
.find(|c: char| !c.is_ascii_digit())
|
||||||
.ok_or("unterminated range selector (missing unit and ']')")?;
|
.ok_or("unterminated range selector (missing ']')")?;
|
||||||
if digits_end == 0 {
|
if digits_end == 0 {
|
||||||
return Err("range selector must start with a number, e.g. [5m]".to_string());
|
return Err("range selector must start with a number, e.g. [5m] or [10]".to_string());
|
||||||
}
|
}
|
||||||
let digits: i64 = rest[..digits_end]
|
let digits: i64 = rest[..digits_end]
|
||||||
.parse()
|
.parse()
|
||||||
.map_err(|_| "range selector duration is not a valid number".to_string())?;
|
.map_err(|_| "range selector number is not a valid number".to_string())?;
|
||||||
self.pos += digits_end;
|
self.pos += digits_end;
|
||||||
|
|
||||||
let unit = self.peek().ok_or("range selector is missing a duration unit")?;
|
if self.peek() == Some(']') {
|
||||||
|
self.expect_char(']')?;
|
||||||
|
return Ok(Some(RangeSelector::BareWidth(digits as f64)));
|
||||||
|
}
|
||||||
|
|
||||||
|
let unit = self.peek().ok_or("range selector is missing a duration unit or ']'")?;
|
||||||
let seconds_per_unit: i64 = match unit {
|
let seconds_per_unit: i64 = match unit {
|
||||||
's' => 1,
|
's' => 1,
|
||||||
'm' => 60,
|
'm' => 60,
|
||||||
@@ -213,14 +254,20 @@ impl<'a> Parser<'a> {
|
|||||||
};
|
};
|
||||||
self.pos += unit.len_utf8();
|
self.pos += unit.len_utf8();
|
||||||
self.expect_char(']')?;
|
self.expect_char(']')?;
|
||||||
Ok(Some(Duration::from_secs((digits * seconds_per_unit).max(0) as u64)))
|
Ok(Some(RangeSelector::Duration(Duration::from_secs(
|
||||||
|
(digits * seconds_per_unit).max(0) as u64,
|
||||||
|
))))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn parse_vector_selector(&mut self) -> Result<PromqlQuery, String> {
|
fn parse_vector_selector(&mut self) -> Result<PromqlQuery, String> {
|
||||||
let metric_name = self.parse_identifier()?;
|
let metric_name = self.parse_identifier()?;
|
||||||
let matchers = self.parse_label_matchers()?;
|
let matchers = self.parse_label_matchers()?;
|
||||||
let range = self.parse_range_selector()?;
|
let (range, bucket_width) = match self.parse_range_selector()? {
|
||||||
Ok(PromqlQuery { agg: None, metric_name, matchers, range })
|
None => (None, None),
|
||||||
|
Some(RangeSelector::Duration(d)) => (Some(d), None),
|
||||||
|
Some(RangeSelector::BareWidth(w)) => (None, Some(w)),
|
||||||
|
};
|
||||||
|
Ok(PromqlQuery { agg: None, metric_name, matchers, range, bucket_width })
|
||||||
}
|
}
|
||||||
|
|
||||||
fn parse_query(&mut self) -> Result<PromqlQuery, String> {
|
fn parse_query(&mut self) -> Result<PromqlQuery, String> {
|
||||||
@@ -231,6 +278,8 @@ impl<'a> Parser<'a> {
|
|||||||
"sum" => Some(AggOp::Sum),
|
"sum" => Some(AggOp::Sum),
|
||||||
"avg" => Some(AggOp::Avg),
|
"avg" => Some(AggOp::Avg),
|
||||||
"rate" => Some(AggOp::Rate),
|
"rate" => Some(AggOp::Rate),
|
||||||
|
"histogram" => Some(AggOp::Histogram),
|
||||||
|
"gauge" => Some(AggOp::Gauge),
|
||||||
_ => None,
|
_ => None,
|
||||||
};
|
};
|
||||||
self.skip_whitespace();
|
self.skip_whitespace();
|
||||||
@@ -366,4 +415,33 @@ mod tests {
|
|||||||
let err = parse_query("requests extra").unwrap_err();
|
let err = parse_query("requests extra").unwrap_err();
|
||||||
assert!(err.contains("unexpected trailing input"));
|
assert!(err.contains("unexpected trailing input"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parses_histogram_with_bare_width() {
|
||||||
|
let query = parse_query("histogram(latency_ms[10])").unwrap();
|
||||||
|
assert_eq!(query.agg, Some(AggOp::Histogram));
|
||||||
|
assert_eq!(query.metric_name, "latency_ms");
|
||||||
|
assert_eq!(query.bucket_width, Some(10.0));
|
||||||
|
assert_eq!(query.range, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_histogram_without_width() {
|
||||||
|
let err = parse_query("histogram(latency_ms)").unwrap_err();
|
||||||
|
assert!(err.contains("bucket width"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parses_gauge_with_no_range_selector() {
|
||||||
|
let query = parse_query(r#"gauge(connections{host="a"})"#).unwrap();
|
||||||
|
assert_eq!(query.agg, Some(AggOp::Gauge));
|
||||||
|
assert_eq!(query.matchers, vec![LabelMatcher { name: "host".to_string(), value: "a".to_string() }]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rate_range_selector_is_still_a_duration_not_a_bare_width() {
|
||||||
|
let query = parse_query("rate(requests[5m])").unwrap();
|
||||||
|
assert_eq!(query.range, Some(Duration::from_secs(300)));
|
||||||
|
assert_eq!(query.bucket_width, None);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -128,9 +128,17 @@ const SECTIONS: DocSection[] = [
|
|||||||
body: (
|
body: (
|
||||||
<div className="flex flex-col gap-2">
|
<div className="flex flex-col gap-2">
|
||||||
<p>
|
<p>
|
||||||
A metric declaration bundles a message-match regex with one or more named param matchers (each with its own
|
Every metric is fundamentally a <em>counter</em>: a message-match regex paired with one or more named param
|
||||||
capture group and type). Every log line matching the regex produces a <em>bucket value</em>: a timestamped
|
matchers (each with its own capture group and type). Every log line matching the regex is one counter
|
||||||
occurrence, tagged with the labels captured from that line.
|
increment — a timestamped occurrence, tagged with the labels captured from that line.
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
Each param is either a <strong>label</strong> (the default) or the metric's single <strong>value</strong>{' '}
|
||||||
|
param. Label params identify which bucket an occurrence belongs to; a value param's captured number becomes
|
||||||
|
that occurrence's stored value instead of a label — e.g. a <Code>duration</Code> param marked{' '}
|
||||||
|
<em>value</em> lets you query the distribution or latest reading of durations, not just how often they
|
||||||
|
occurred. A metric with only label params still works exactly as before: every occurrence's value defaults
|
||||||
|
to a fixed <Code>1</Code> (a pure occurrence count).
|
||||||
</p>
|
</p>
|
||||||
<p>
|
<p>
|
||||||
Declaring a metric backfills it against everything already ingested, then continues indexing new lines as
|
Declaring a metric backfills it against everything already ingested, then continues indexing new lines as
|
||||||
@@ -139,8 +147,25 @@ const SECTIONS: DocSection[] = [
|
|||||||
<p>
|
<p>
|
||||||
The Metrics panel lets you declare a metric, pick one from the declared list, and see its bucket values
|
The Metrics panel lets you declare a metric, pick one from the declared list, and see its bucket values
|
||||||
grouped by label combination over the active time range — occurrence count per group, plus a small trend
|
grouped by label combination over the active time range — occurrence count per group, plus a small trend
|
||||||
sparkline. That same data is what dashboard panels query via PromQL (see below).
|
sparkline. That same data is what dashboard panels query via PromQL (see below), including two forms{' '}
|
||||||
|
<em>derived</em> from a counter's value param rather than stored separately:
|
||||||
</p>
|
</p>
|
||||||
|
<ul className="list-disc pl-5">
|
||||||
|
<li>
|
||||||
|
<span className="font-semibold text-text">
|
||||||
|
<Code>histogram(metric_name[width])</Code>
|
||||||
|
</span>{' '}
|
||||||
|
— buckets every matching event's value into fixed-width buckets of size <Code>width</Code>, e.g.{' '}
|
||||||
|
<Code>histogram(durations[10])</Code>.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<span className="font-semibold text-text">
|
||||||
|
<Code>gauge(metric_name)</Code>
|
||||||
|
</span>{' '}
|
||||||
|
— the most-recent value per label combination in the active range, i.e. the counter collapsed to its
|
||||||
|
latest reading.
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
</div>
|
</div>
|
||||||
),
|
),
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -214,6 +214,15 @@ export function Metrics({ range }: { range: TimeRange | null }) {
|
|||||||
<option value="number">number</option>
|
<option value="number">number</option>
|
||||||
<option value="timestamp">timestamp</option>
|
<option value="timestamp">timestamp</option>
|
||||||
</select>
|
</select>
|
||||||
|
<select
|
||||||
|
className="rounded border border-border-soft bg-surface-1 px-1 py-1 text-xs text-text focus:border-border-strong focus:outline-none"
|
||||||
|
value={matcher.role ?? 'label'}
|
||||||
|
onChange={(e) => updateMatcher(i, { role: e.target.value as MetricParamMatcher['role'] })}
|
||||||
|
title="Label params identify a bucket; at most one Value param becomes the counter's per-event captured value (used by histogram()/gauge())"
|
||||||
|
>
|
||||||
|
<option value="label">label</option>
|
||||||
|
<option value="value">value</option>
|
||||||
|
</select>
|
||||||
<button
|
<button
|
||||||
type="button"
|
type="button"
|
||||||
onClick={() => removeMatcher(i)}
|
onClick={() => removeMatcher(i)}
|
||||||
|
|||||||
@@ -43,10 +43,15 @@ export interface Stats {
|
|||||||
last_timestamp_ms: number | null
|
last_timestamp_ms: number | null
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Whether a param contributes to a bucket's labels or is its captured value (ticket 022). */
|
||||||
|
export type ParamRole = 'label' | 'value'
|
||||||
|
|
||||||
export interface MetricParamMatcher {
|
export interface MetricParamMatcher {
|
||||||
param_name: string
|
param_name: string
|
||||||
capture_group: string
|
capture_group: string
|
||||||
kind: 'string' | 'number' | 'timestamp'
|
kind: 'string' | 'number' | 'timestamp'
|
||||||
|
/** Defaults to 'label' server-side when omitted. */
|
||||||
|
role?: ParamRole
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface MetricDeclaration {
|
export interface MetricDeclaration {
|
||||||
|
|||||||
Reference in New Issue
Block a user