Add alert rules with FilterBar-based triggers and ntfy delivery (tickets 017-019)
Adds AlertRule/AlertCondition/AlertSink models with CRUD persisted on Settings (mirroring CssEntry's shape), per-line trigger evaluation hooked into the engine's ingest path (Immediate/Threshold/NewValue/Ratio conditions), and best-effort ntfy delivery spawned off the ingest hot path. Design notes: - Alert queries reuse FilterBar syntax but only support the regex()/sequence/timestamp subset (no metric[param] comparisons), since resolving those would require a full bucket rescan per alert per ingested line rather than cost proportional to new logs. - Ratio's zero-denominator case skips evaluation for that line rather than treating the ratio as infinite. - NewValue's seen-value set is in-memory only and does not survive a restart, matching the engine's other non-persisted runtime state. engine::run now takes a SettingsHandle (moved from api.rs to settings.rs so engine.rs can depend on it without a circular import) to read the current alert rules on each ingest. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
758feca780
commit
dd3320812c
+654
@@ -0,0 +1,654 @@
|
||||
//! Alert rules: definitions (ticket 017), trigger evaluation (ticket 018),
|
||||
//! and ntfy delivery (ticket 019).
|
||||
//!
|
||||
//! ## Design: why alert queries only support regex/sequence/timestamp
|
||||
//!
|
||||
//! `Immediate`/`Threshold`/`Ratio` conditions reuse FilterBar query strings
|
||||
//! (see `crate::filter_parser`), evaluated against a *single newly-ingested
|
||||
//! line* on `EngineMessage::Ingest` (see `evaluate_alerts`, called from
|
||||
//! `engine::run`). `engine.rs`'s own `compile_filter_query` resolves
|
||||
//! `metric[param]` comparisons by scanning that metric's *entire* bucket
|
||||
//! history up front (`compile_metric_comparison`) — reusing that per alert
|
||||
//! check, per ingested line, would make alert evaluation cost proportional to
|
||||
//! total log volume rather than to new logs ingested, which this ticket
|
||||
//! requires. So alert queries here only support the subset of FilterBar that
|
||||
//! evaluates against a single line's own fields — `regex(...)`, `sequence`,
|
||||
//! `timestamp`, and `and`/`or` — via `eval_alert_expr` below. A `metric[param]`
|
||||
//! comparison in an alert query is rejected with a clear error (logged, the
|
||||
//! rule just never fires) rather than silently misbehaving.
|
||||
|
||||
use std::collections::hash_map::DefaultHasher;
|
||||
use std::collections::{HashMap, HashSet, VecDeque};
|
||||
use std::fmt;
|
||||
use std::hash::{Hash, Hasher};
|
||||
use std::io;
|
||||
use std::path::Path;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use regex::Regex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::engine::apply_compare_op;
|
||||
use crate::filter_datetime;
|
||||
use crate::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
|
||||
use crate::settings::Settings;
|
||||
use crate::timestamp::Timestamp;
|
||||
|
||||
/// A single alert rule definition (ticket 017).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct AlertRule {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub enabled: bool,
|
||||
pub condition: AlertCondition,
|
||||
pub sink: AlertSink,
|
||||
}
|
||||
|
||||
/// What triggers an alert rule (ticket 018).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(tag = "kind", rename_all = "lowercase")]
|
||||
pub enum AlertCondition {
|
||||
/// Fires once per log line matching `query`.
|
||||
Immediate { query: String },
|
||||
/// Fires when `query`'s match count within a rolling `window_ms` crosses
|
||||
/// `threshold` (via `op`).
|
||||
Threshold {
|
||||
query: String,
|
||||
op: CompareOp,
|
||||
threshold: f64,
|
||||
window_ms: i64,
|
||||
},
|
||||
/// Fires the first time `field_regex` matches a line and `capture_group`
|
||||
/// captures a value not seen before *for this rule*. Seen values live
|
||||
/// only in memory (`AlertRuntimeState::seen_values`) for the process's
|
||||
/// lifetime — a restart forgets them, so previously-seen values can fire
|
||||
/// again. This is a deliberate simplicity tradeoff (matches the
|
||||
/// codebase's general "no restart persistence for in-memory engine
|
||||
/// state" pattern), not an oversight.
|
||||
NewValue {
|
||||
field_regex: String,
|
||||
capture_group: String,
|
||||
},
|
||||
/// Fires when `count(query_a) / count(query_b)` (rolling over
|
||||
/// `window_ms`) crosses `ratio` (via `op`). A zero `count_b` skips
|
||||
/// evaluation entirely for that line (rather than treating the ratio as
|
||||
/// infinite) — there's no meaningful "actual vs threshold" comparison to
|
||||
/// report yet.
|
||||
Ratio {
|
||||
query_a: String,
|
||||
query_b: String,
|
||||
op: CompareOp,
|
||||
ratio: f64,
|
||||
window_ms: i64,
|
||||
},
|
||||
}
|
||||
|
||||
/// Where a fired alert is delivered (ticket 019). Modeled as an enum (rather
|
||||
/// than `AlertRule` holding a bare topic/server) so more sink kinds can be
|
||||
/// added later without reshaping `AlertRule`.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(tag = "kind", rename_all = "lowercase")]
|
||||
pub enum AlertSink {
|
||||
Ntfy {
|
||||
topic: String,
|
||||
/// Defaults to `https://ntfy.sh` when absent.
|
||||
server: Option<String>,
|
||||
},
|
||||
}
|
||||
|
||||
/// Error returned by alert-rule mutations/lookups. Mirrors `CssError`'s shape.
|
||||
#[derive(Debug)]
|
||||
pub enum AlertError {
|
||||
NotFound,
|
||||
Io(io::Error),
|
||||
}
|
||||
|
||||
impl fmt::Display for AlertError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
AlertError::NotFound => write!(f, "no alert rule with that id"),
|
||||
AlertError::Io(err) => write!(f, "I/O error: {err}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for AlertError {}
|
||||
|
||||
impl From<io::Error> for AlertError {
|
||||
fn from(err: io::Error) -> Self {
|
||||
AlertError::Io(err)
|
||||
}
|
||||
}
|
||||
|
||||
/// Generates an id for a newly-created alert rule. Same approach as
|
||||
/// `settings::generate_imported_css_id`: hash the name and current time
|
||||
/// rather than pulling in a `uuid` dependency.
|
||||
fn generate_alert_rule_id(name: &str) -> String {
|
||||
let mut hasher = DefaultHasher::new();
|
||||
name.hash(&mut hasher);
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
nanos.hash(&mut hasher);
|
||||
format!("alert-{:016x}", hasher.finish())
|
||||
}
|
||||
|
||||
impl Settings {
|
||||
/// Creates a new alert rule, persisting settings to `settings_path`.
|
||||
pub fn create_alert_rule(
|
||||
&mut self,
|
||||
settings_path: &Path,
|
||||
name: String,
|
||||
enabled: bool,
|
||||
condition: AlertCondition,
|
||||
sink: AlertSink,
|
||||
) -> Result<AlertRule, AlertError> {
|
||||
let rule = AlertRule { id: generate_alert_rule_id(&name), name, enabled, condition, sink };
|
||||
self.alert_rules.push(rule.clone());
|
||||
self.save(settings_path)?;
|
||||
Ok(rule)
|
||||
}
|
||||
|
||||
/// Replaces an existing alert rule's fields (id stays fixed), persisting
|
||||
/// settings to `settings_path`. Returns `AlertError::NotFound` if no rule
|
||||
/// with `id` exists.
|
||||
pub fn update_alert_rule(
|
||||
&mut self,
|
||||
settings_path: &Path,
|
||||
id: &str,
|
||||
name: String,
|
||||
enabled: bool,
|
||||
condition: AlertCondition,
|
||||
sink: AlertSink,
|
||||
) -> Result<AlertRule, AlertError> {
|
||||
let rule = self
|
||||
.alert_rules
|
||||
.iter_mut()
|
||||
.find(|rule| rule.id == id)
|
||||
.ok_or(AlertError::NotFound)?;
|
||||
rule.name = name;
|
||||
rule.enabled = enabled;
|
||||
rule.condition = condition;
|
||||
rule.sink = sink;
|
||||
let updated = rule.clone();
|
||||
self.save(settings_path)?;
|
||||
Ok(updated)
|
||||
}
|
||||
|
||||
/// Removes an alert rule by id, persisting settings to `settings_path`.
|
||||
/// Returns `AlertError::NotFound` if no rule with `id` exists.
|
||||
pub fn remove_alert_rule(&mut self, settings_path: &Path, id: &str) -> Result<(), AlertError> {
|
||||
let index = self
|
||||
.alert_rules
|
||||
.iter()
|
||||
.position(|rule| rule.id == id)
|
||||
.ok_or(AlertError::NotFound)?;
|
||||
self.alert_rules.remove(index);
|
||||
self.save(settings_path)?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-rule in-memory evaluation state (ticket 018): rolling match-timestamp
|
||||
/// windows for `Threshold`/`Ratio`, and seen-values for `NewValue`. Lives
|
||||
/// only for the engine process's lifetime (see `AlertCondition::NewValue`'s
|
||||
/// doc comment) — not persisted, not backfilled against already-ingested
|
||||
/// logs.
|
||||
#[derive(Debug, Default)]
|
||||
pub struct AlertRuntimeState {
|
||||
threshold_matches: VecDeque<i64>,
|
||||
ratio_a_matches: VecDeque<i64>,
|
||||
ratio_b_matches: VecDeque<i64>,
|
||||
seen_values: HashSet<String>,
|
||||
}
|
||||
|
||||
/// Drops every entry older than `window_ms` relative to `now_ms` from the
|
||||
/// front of a rolling window. `timestamps` is push_back-ordered, so once a
|
||||
/// front entry is within the window every entry behind it is too.
|
||||
fn prune_window(timestamps: &mut VecDeque<i64>, now_ms: i64, window_ms: i64) {
|
||||
while let Some(&front) = timestamps.front() {
|
||||
if now_ms - front > window_ms {
|
||||
timestamps.pop_front();
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Evaluates a FilterBar query string against a single log line's own fields
|
||||
/// only — see this module's doc comment for why `metric[param]` comparisons
|
||||
/// aren't supported here.
|
||||
fn eval_alert_query(query: &str, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
|
||||
let expr = filter_parser::parse(query).map_err(|err| err.to_string())?;
|
||||
eval_alert_expr(&expr, sequence, timestamp, text)
|
||||
}
|
||||
|
||||
fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
|
||||
match expr {
|
||||
FilterExpr::Regex(pattern) => {
|
||||
let re = Regex::new(pattern).map_err(|err| format!("invalid regex in alert query: {err}"))?;
|
||||
Ok(re.is_match(text))
|
||||
}
|
||||
FilterExpr::Comparison { metric, param: None, op, value } if metric == "sequence" => {
|
||||
let target = match value {
|
||||
FilterValue::Number(n) => *n,
|
||||
FilterValue::Raw(raw) => raw
|
||||
.parse::<f64>()
|
||||
.map_err(|_| format!("'sequence' comparisons require a numeric value, got {raw:?}"))?,
|
||||
};
|
||||
Ok(apply_compare_op((sequence as f64).total_cmp(&target), *op))
|
||||
}
|
||||
FilterExpr::Comparison { metric, param: None, op, value } if metric == "timestamp" => {
|
||||
let target_ms = match value {
|
||||
FilterValue::Number(n) => *n as i64,
|
||||
FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, Timestamp::now())
|
||||
.map_err(|err| err.to_string())?
|
||||
.as_millis(),
|
||||
};
|
||||
Ok(apply_compare_op(timestamp.as_millis().cmp(&target_ms), *op))
|
||||
}
|
||||
FilterExpr::Comparison { metric, .. } => Err(format!(
|
||||
"alert queries only support regex()/sequence/timestamp (not metric comparisons like \
|
||||
'{metric}[param]'): metric-based alert conditions aren't supported yet"
|
||||
)),
|
||||
FilterExpr::And(lhs, rhs) => {
|
||||
Ok(eval_alert_expr(lhs, sequence, timestamp, text)? && eval_alert_expr(rhs, sequence, timestamp, text)?)
|
||||
}
|
||||
FilterExpr::Or(lhs, rhs) => {
|
||||
Ok(eval_alert_expr(lhs, sequence, timestamp, text)? || eval_alert_expr(rhs, sequence, timestamp, text)?)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Evaluates every enabled rule's condition against one newly-ingested line,
|
||||
/// mutating each rule's runtime state (rolling windows / seen-values) along
|
||||
/// the way. Returns `(rule, message)` for every rule that fired — the caller
|
||||
/// (`engine::run`) hands each off to `deliver_for_rule`. A malformed query
|
||||
/// (bad regex, unsupported metric comparison) is logged to stderr and simply
|
||||
/// never matches, rather than panicking or propagating.
|
||||
pub fn evaluate_alerts(
|
||||
rules: &[AlertRule],
|
||||
states: &mut HashMap<String, AlertRuntimeState>,
|
||||
sequence: u64,
|
||||
timestamp: Timestamp,
|
||||
text: &str,
|
||||
) -> Vec<(AlertRule, String)> {
|
||||
let mut fired = Vec::new();
|
||||
for rule in rules {
|
||||
if !rule.enabled {
|
||||
continue;
|
||||
}
|
||||
let state = states.entry(rule.id.clone()).or_default();
|
||||
if let Some(message) = evaluate_condition(&rule.condition, state, sequence, timestamp, text, &rule.name) {
|
||||
fired.push((rule.clone(), message));
|
||||
}
|
||||
}
|
||||
fired
|
||||
}
|
||||
|
||||
fn evaluate_condition(
|
||||
condition: &AlertCondition,
|
||||
state: &mut AlertRuntimeState,
|
||||
sequence: u64,
|
||||
timestamp: Timestamp,
|
||||
text: &str,
|
||||
rule_name: &str,
|
||||
) -> Option<String> {
|
||||
match condition {
|
||||
AlertCondition::Immediate { query } => match eval_alert_query(query, sequence, timestamp, text) {
|
||||
Ok(true) => Some(format!("{rule_name}: matched line: {text}")),
|
||||
Ok(false) => None,
|
||||
Err(err) => {
|
||||
eprintln!("warning: alert '{rule_name}' query error: {err}");
|
||||
None
|
||||
}
|
||||
},
|
||||
AlertCondition::Threshold { query, op, threshold, window_ms } => {
|
||||
match eval_alert_query(query, sequence, timestamp, text) {
|
||||
Ok(true) => {
|
||||
let now_ms = timestamp.as_millis();
|
||||
state.threshold_matches.push_back(now_ms);
|
||||
prune_window(&mut state.threshold_matches, now_ms, *window_ms);
|
||||
let count = state.threshold_matches.len() as f64;
|
||||
apply_compare_op(count.total_cmp(threshold), *op).then(|| {
|
||||
format!("{rule_name}: match count {count} {op:?} threshold {threshold} (window {window_ms}ms)")
|
||||
})
|
||||
}
|
||||
Ok(false) => None,
|
||||
Err(err) => {
|
||||
eprintln!("warning: alert '{rule_name}' query error: {err}");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
AlertCondition::NewValue { field_regex, capture_group } => {
|
||||
let re = match Regex::new(field_regex) {
|
||||
Ok(re) => re,
|
||||
Err(err) => {
|
||||
eprintln!("warning: alert '{rule_name}' has an invalid field_regex: {err}");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let captures = re.captures(text)?;
|
||||
let value = captures.name(capture_group)?.as_str().to_string();
|
||||
state
|
||||
.seen_values
|
||||
.insert(value.clone())
|
||||
.then(|| format!("{rule_name}: new value seen: {value} (line: {text})"))
|
||||
}
|
||||
AlertCondition::Ratio { query_a, query_b, op, ratio, window_ms } => {
|
||||
let matched_a = eval_alert_query(query_a, sequence, timestamp, text);
|
||||
let matched_b = eval_alert_query(query_b, sequence, timestamp, text);
|
||||
let (matched_a, matched_b) = match (matched_a, matched_b) {
|
||||
(Ok(a), Ok(b)) => (a, b),
|
||||
(Err(err), _) | (_, Err(err)) => {
|
||||
eprintln!("warning: alert '{rule_name}' query error: {err}");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
if !matched_a && !matched_b {
|
||||
return None;
|
||||
}
|
||||
let now_ms = timestamp.as_millis();
|
||||
if matched_a {
|
||||
state.ratio_a_matches.push_back(now_ms);
|
||||
prune_window(&mut state.ratio_a_matches, now_ms, *window_ms);
|
||||
}
|
||||
if matched_b {
|
||||
state.ratio_b_matches.push_back(now_ms);
|
||||
prune_window(&mut state.ratio_b_matches, now_ms, *window_ms);
|
||||
}
|
||||
let count_a = state.ratio_a_matches.len() as f64;
|
||||
let count_b = state.ratio_b_matches.len() as f64;
|
||||
if count_b == 0.0 {
|
||||
// Zero denominator: skip evaluation rather than treat the
|
||||
// ratio as infinite — there's nothing meaningful to compare
|
||||
// against `ratio` yet.
|
||||
return None;
|
||||
}
|
||||
let observed = count_a / count_b;
|
||||
apply_compare_op(observed.total_cmp(ratio), *op).then(|| {
|
||||
format!(
|
||||
"{rule_name}: ratio {observed:.3} ({count_a}/{count_b}) {op:?} {ratio} (window {window_ms}ms)"
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Delivers one fired alert to its rule's configured sink. Errors are logged
|
||||
/// to stderr and dropped — no retry queue, matching `remove_css`'s
|
||||
/// best-effort style. Called via `tokio::spawn` from the ingest hot path
|
||||
/// (`engine::run`) so a slow/unreachable sink never blocks ingestion.
|
||||
pub async fn deliver_for_rule(rule: AlertRule, message: String) {
|
||||
match rule.sink {
|
||||
AlertSink::Ntfy { topic, server } => {
|
||||
let server = server.unwrap_or_else(|| "https://ntfy.sh".to_string());
|
||||
if let Err(err) = deliver_ntfy(&server, &topic, &message).await {
|
||||
eprintln!("warning: failed to deliver alert '{}' via ntfy: {err}", rule.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Publishes `message` to `{server}/{topic}` via ntfy's plain-text publish
|
||||
/// API (no auth, raw body). Standalone and `AlertRule`-free so a future
|
||||
/// "send test notification" endpoint can call it directly.
|
||||
pub async fn deliver_ntfy(server: &str, topic: &str, message: &str) -> Result<(), String> {
|
||||
let url = format!("{}/{}", server.trim_end_matches('/'), topic);
|
||||
let response = reqwest::Client::new()
|
||||
.post(&url)
|
||||
.body(message.to_string())
|
||||
.send()
|
||||
.await
|
||||
.map_err(|err| err.to_string())?;
|
||||
if response.status().is_success() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!("ntfy responded with status {}", response.status()))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn line(text: &str, sequence: u64, timestamp_ms: i64) -> (u64, Timestamp, &str) {
|
||||
(sequence, Timestamp::from_millis(timestamp_ms), text)
|
||||
}
|
||||
|
||||
fn immediate_rule(query: &str) -> AlertRule {
|
||||
AlertRule {
|
||||
id: "r1".to_string(),
|
||||
name: "test rule".to_string(),
|
||||
enabled: true,
|
||||
condition: AlertCondition::Immediate { query: query.to_string() },
|
||||
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn alert_rule_crud_round_trips_through_save_and_load() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let settings_path = dir.path().join("settings.json");
|
||||
let mut settings = Settings::load(&settings_path);
|
||||
|
||||
let created = settings
|
||||
.create_alert_rule(
|
||||
&settings_path,
|
||||
"spike".to_string(),
|
||||
true,
|
||||
AlertCondition::Immediate { query: r#"regex("error")"#.to_string() },
|
||||
AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(settings.alert_rules.len(), 1);
|
||||
|
||||
let reloaded = Settings::load(&settings_path);
|
||||
assert_eq!(reloaded.alert_rules, settings.alert_rules);
|
||||
|
||||
let updated = settings
|
||||
.update_alert_rule(
|
||||
&settings_path,
|
||||
&created.id,
|
||||
"renamed".to_string(),
|
||||
false,
|
||||
AlertCondition::Immediate { query: r#"regex("warn")"#.to_string() },
|
||||
AlertSink::Ntfy { topic: "other".to_string(), server: Some("https://example.com".to_string()) },
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(updated.id, created.id);
|
||||
assert_eq!(updated.name, "renamed");
|
||||
assert!(!updated.enabled);
|
||||
|
||||
settings.remove_alert_rule(&settings_path, &created.id).unwrap();
|
||||
assert!(settings.alert_rules.is_empty());
|
||||
let reloaded = Settings::load(&settings_path);
|
||||
assert!(reloaded.alert_rules.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn alert_rule_mutations_reject_unknown_id() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let settings_path = dir.path().join("settings.json");
|
||||
let mut settings = Settings::load(&settings_path);
|
||||
|
||||
let result = settings.update_alert_rule(
|
||||
&settings_path,
|
||||
"does-not-exist",
|
||||
"x".to_string(),
|
||||
true,
|
||||
AlertCondition::Immediate { query: r#"regex("x")"#.to_string() },
|
||||
AlertSink::Ntfy { topic: "x".to_string(), server: None },
|
||||
);
|
||||
assert!(matches!(result, Err(AlertError::NotFound)));
|
||||
|
||||
let result = settings.remove_alert_rule(&settings_path, "does-not-exist");
|
||||
assert!(matches!(result, Err(AlertError::NotFound)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn immediate_fires_on_matching_line_only() {
|
||||
let rules = vec![immediate_rule(r#"regex("error")"#)];
|
||||
let mut states = HashMap::new();
|
||||
|
||||
let (seq, ts, text) = line("all good", 0, 0);
|
||||
assert!(evaluate_alerts(&rules, &mut states, seq, ts, text).is_empty());
|
||||
|
||||
let (seq, ts, text) = line("an error happened", 1, 1);
|
||||
let fired = evaluate_alerts(&rules, &mut states, seq, ts, text);
|
||||
assert_eq!(fired.len(), 1);
|
||||
assert!(fired[0].1.contains("an error happened"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn disabled_rule_never_fires() {
|
||||
let mut rule = immediate_rule(r#"regex("error")"#);
|
||||
rule.enabled = false;
|
||||
let mut states = HashMap::new();
|
||||
let (seq, ts, text) = line("error!", 0, 0);
|
||||
assert!(evaluate_alerts(&[rule], &mut states, seq, ts, text).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn threshold_fires_once_count_crosses_and_not_before() {
|
||||
let rule = AlertRule {
|
||||
id: "r1".to_string(),
|
||||
name: "spike".to_string(),
|
||||
enabled: true,
|
||||
condition: AlertCondition::Threshold {
|
||||
query: r#"regex("error")"#.to_string(),
|
||||
op: CompareOp::Ge,
|
||||
threshold: 3.0,
|
||||
window_ms: 10_000,
|
||||
},
|
||||
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
|
||||
};
|
||||
let mut states = HashMap::new();
|
||||
for (seq, ms) in [(0u64, 0i64), (1, 1000)] {
|
||||
let fired = evaluate_alerts(&[rule.clone()], &mut states, seq, Timestamp::from_millis(ms), "error");
|
||||
assert!(fired.is_empty(), "should not fire before threshold is crossed");
|
||||
}
|
||||
let fired = evaluate_alerts(&[rule.clone()], &mut states, 2, Timestamp::from_millis(2000), "error");
|
||||
assert_eq!(fired.len(), 1);
|
||||
// Stays quiet for a non-matching line even though the window is full.
|
||||
let fired = evaluate_alerts(&[rule], &mut states, 3, Timestamp::from_millis(2100), "fine");
|
||||
assert!(fired.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn threshold_window_expires_old_matches() {
|
||||
let rule = AlertRule {
|
||||
id: "r1".to_string(),
|
||||
name: "spike".to_string(),
|
||||
enabled: true,
|
||||
condition: AlertCondition::Threshold {
|
||||
query: r#"regex("error")"#.to_string(),
|
||||
op: CompareOp::Ge,
|
||||
threshold: 2.0,
|
||||
window_ms: 1_000,
|
||||
},
|
||||
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
|
||||
};
|
||||
let mut states = HashMap::new();
|
||||
assert!(evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "error").is_empty());
|
||||
// Second match arrives well after the first has aged out of the window.
|
||||
assert!(evaluate_alerts(&[rule], &mut states, 1, Timestamp::from_millis(5_000), "error").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn new_value_fires_only_on_first_sighting() {
|
||||
let rule = AlertRule {
|
||||
id: "r1".to_string(),
|
||||
name: "new user agent".to_string(),
|
||||
enabled: true,
|
||||
condition: AlertCondition::NewValue {
|
||||
field_regex: r"agent=(?P<agent>\S+)".to_string(),
|
||||
capture_group: "agent".to_string(),
|
||||
},
|
||||
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
|
||||
};
|
||||
let mut states = HashMap::new();
|
||||
let fired = evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "agent=curl");
|
||||
assert_eq!(fired.len(), 1);
|
||||
let fired = evaluate_alerts(&[rule.clone()], &mut states, 1, Timestamp::from_millis(1), "agent=curl");
|
||||
assert!(fired.is_empty());
|
||||
let fired = evaluate_alerts(&[rule], &mut states, 2, Timestamp::from_millis(2), "agent=wget");
|
||||
assert_eq!(fired.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ratio_fires_when_crossing_and_skips_zero_denominator() {
|
||||
let rule = AlertRule {
|
||||
id: "r1".to_string(),
|
||||
name: "error ratio".to_string(),
|
||||
enabled: true,
|
||||
condition: AlertCondition::Ratio {
|
||||
query_a: r#"regex("error")"#.to_string(),
|
||||
query_b: r#"regex("request")"#.to_string(),
|
||||
op: CompareOp::Ge,
|
||||
ratio: 0.5,
|
||||
window_ms: 10_000,
|
||||
},
|
||||
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
|
||||
};
|
||||
let mut states = HashMap::new();
|
||||
// Only an "error" so far; no "request" yet, so denominator is 0 -> skip.
|
||||
assert!(evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "error").is_empty());
|
||||
// One request: ratio 1/1 = 1.0 >= 0.5 -> fires.
|
||||
let fired = evaluate_alerts(&[rule.clone()], &mut states, 1, Timestamp::from_millis(1), "request");
|
||||
assert_eq!(fired.len(), 1);
|
||||
// A second request: ratio 1/2 = 0.5 still >= 0.5 -> fires again.
|
||||
let fired = evaluate_alerts(&[rule.clone()], &mut states, 2, Timestamp::from_millis(2), "request");
|
||||
assert_eq!(fired.len(), 1);
|
||||
// A third request with no more errors: ratio drops to 1/3 -> no longer fires.
|
||||
let fired = evaluate_alerts(&[rule], &mut states, 3, Timestamp::from_millis(3), "request");
|
||||
assert!(fired.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metric_comparison_in_alert_query_is_rejected_not_panicking() {
|
||||
let rule = immediate_rule("some_metric[param]==5");
|
||||
let mut states = HashMap::new();
|
||||
// Should log a warning and simply not fire, never panic.
|
||||
let fired = evaluate_alerts(&[rule], &mut states, 0, Timestamp::from_millis(0), "anything");
|
||||
assert!(fired.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deliver_ntfy_posts_message_to_server_topic_path() {
|
||||
use axum::extract::State;
|
||||
use axum::routing::post;
|
||||
use axum::Router;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use tokio::net::TcpListener;
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct Captured(Arc<Mutex<Option<(String, String)>>>);
|
||||
|
||||
async fn capture(
|
||||
State(captured): State<Captured>,
|
||||
axum::extract::Path(topic): axum::extract::Path<String>,
|
||||
body: String,
|
||||
) -> &'static str {
|
||||
*captured.0.lock().unwrap() = Some((topic, body));
|
||||
"ok"
|
||||
}
|
||||
|
||||
let captured = Captured::default();
|
||||
let app = Router::new().route("/{topic}", post(capture)).with_state(captured.clone());
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
axum::serve(listener, app).await.unwrap();
|
||||
});
|
||||
|
||||
let server = format!("http://{addr}");
|
||||
deliver_ntfy(&server, "alerts", "hello world").await.unwrap();
|
||||
|
||||
let (topic, body) = captured.0.lock().unwrap().clone().unwrap();
|
||||
assert_eq!(topic, "alerts");
|
||||
assert_eq!(body, "hello world");
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user