Add alert rules with FilterBar-based triggers and ntfy delivery (tickets 017-019)

Adds AlertRule/AlertCondition/AlertSink models with CRUD persisted on
Settings (mirroring CssEntry's shape), per-line trigger evaluation hooked
into the engine's ingest path (Immediate/Threshold/NewValue/Ratio
conditions), and best-effort ntfy delivery spawned off the ingest hot path.

Design notes:
- Alert queries reuse FilterBar syntax but only support the
  regex()/sequence/timestamp subset (no metric[param] comparisons), since
  resolving those would require a full bucket rescan per alert per ingested
  line rather than cost proportional to new logs.
- Ratio's zero-denominator case skips evaluation for that line rather than
  treating the ratio as infinite.
- NewValue's seen-value set is in-memory only and does not survive a
  restart, matching the engine's other non-persisted runtime state.

engine::run now takes a SettingsHandle (moved from api.rs to settings.rs so
engine.rs can depend on it without a circular import) to read the current
alert rules on each ingest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
Hugh Kaznowski
2026-09-05 16:30:14 +01:00
co-authored by Claude Sonnet 5
parent 758feca780
commit dd3320812c
13 changed files with 1579 additions and 180 deletions
+20 -3
View File
@@ -1,3 +1,4 @@
use crate::alerts::{self, AlertRuntimeState};
use crate::data_value::{DataValue, MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher};
use crate::dto::{
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
@@ -8,11 +9,12 @@ use crate::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::index_key::IndexKey;
use crate::metric_primitive::{MetricPrimitive, Number};
use crate::promql::{self, AggOp, LabelMatcher, PromqlQuery};
use crate::settings::SettingsHandle;
use crate::timestamp::Timestamp;
use regex::Regex;
use serde::Deserialize;
use std::cmp::Ordering;
use std::collections::{BTreeMap, HashSet};
use std::collections::{BTreeMap, HashMap, HashSet};
use tokio::sync::{mpsc, oneshot};
pub struct IngestedLine {
@@ -157,10 +159,14 @@ pub type EngineHandle = mpsc::Sender<EngineMessage>;
/// Sole owner of the primary index. Runs single-threaded so the map never needs a
/// lock, and does its own response serialization/DTO construction so callers never
/// have to clone the index to answer a query.
pub async fn run(mut rx: mpsc::Receiver<EngineMessage>) {
pub async fn run(mut rx: mpsc::Receiver<EngineMessage>, settings: SettingsHandle) {
let mut index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let mut max_logs: Option<usize> = None;
let mut log_count: usize = 0;
// Ticket 018: per-rule rolling windows / seen-value sets, keyed by rule
// id. Lives only as long as this loop (no restart persistence — see
// `alerts::AlertCondition::NewValue`'s doc comment).
let mut alert_states: HashMap<String, AlertRuntimeState> = HashMap::new();
while let Some(message) = rx.recv().await {
match message {
@@ -173,6 +179,15 @@ pub async fn run(mut rx: mpsc::Receiver<EngineMessage>) {
log_count += 1;
apply_all_metrics(&mut index, sequence, timestamp, &text);
evict_excess_logs(&mut index, &mut log_count, max_logs);
// Ticket 018: evaluate alert rules against just this line
// (not the whole log range), then hand any fired rule off to
// ticket 019's delivery, off the hot path via `tokio::spawn`.
let rules = settings.lock().unwrap().alert_rules.clone();
let fired = alerts::evaluate_alerts(&rules, &mut alert_states, sequence, timestamp, &text);
for (rule, message) in fired {
tokio::spawn(alerts::deliver_for_rule(rule, message));
}
}
EngineMessage::SetMaxLogs(new_max) => {
max_logs = new_max;
@@ -739,7 +754,9 @@ fn compare_metric_primitive(
Ok(ordering.is_some_and(|ordering| apply_compare_op(ordering, op)))
}
fn apply_compare_op(ordering: Ordering, op: CompareOp) -> bool {
/// `pub(crate)` so `crate::alerts` can reuse it for `Threshold`/`Ratio`
/// comparisons instead of duplicating the match.
pub(crate) fn apply_compare_op(ordering: Ordering, op: CompareOp) -> bool {
match op {
CompareOp::Lt => ordering == Ordering::Less,
CompareOp::Gt => ordering == Ordering::Greater,