Add alert rules with FilterBar-based triggers and ntfy delivery (tickets 017-019)

Adds AlertRule/AlertCondition/AlertSink models with CRUD persisted on
Settings (mirroring CssEntry's shape), per-line trigger evaluation hooked
into the engine's ingest path (Immediate/Threshold/NewValue/Ratio
conditions), and best-effort ntfy delivery spawned off the ingest hot path.

Design notes:
- Alert queries reuse FilterBar syntax but only support the
  regex()/sequence/timestamp subset (no metric[param] comparisons), since
  resolving those would require a full bucket rescan per alert per ingested
  line rather than cost proportional to new logs.
- Ratio's zero-denominator case skips evaluation for that line rather than
  treating the ratio as infinite.
- NewValue's seen-value set is in-memory only and does not survive a
  restart, matching the engine's other non-persisted runtime state.

engine::run now takes a SettingsHandle (moved from api.rs to settings.rs so
engine.rs can depend on it without a circular import) to read the current
alert rules on each ingest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
Hugh Kaznowski
2026-09-05 16:30:14 +01:00
co-authored by Claude Sonnet 5
parent 758feca780
commit dd3320812c
13 changed files with 1579 additions and 180 deletions
-49
View File
@@ -1,49 +0,0 @@
# 017 — Alert definitions & settings persistence
## Summary
Foundation for the alerting feature: a persisted `AlertRule` model plus CRUD,
following the shape `Settings` already uses for `CssEntry`/`Profiles`
([008](008-settings-model.md)) — one process-wide list, saved to disk on every
mutation.
An `AlertRule` has:
- A **condition** (see [018](018-alert-trigger-evaluation.md) for the types and
how they're evaluated) — for this ticket, just store it as an opaque, already
parsed/validated value; don't implement evaluation here.
- A **sink**: for now, only `Ntfy { topic: String, server: Option<String> }`
(`server` defaults to `https://ntfy.sh` when absent) — modeled as an enum so
[additional sinks](020-alerts-webapp-ui.md) can be added later without
reshaping `AlertRule`.
- Bookkeeping: `id`, `name`, `enabled: bool`.
## Scope
- `AlertRule` struct (or similar) in a new `src/alerts.rs`, `Vec<AlertRule>`
added to `Settings` (`src/settings.rs`), persisted the same way
`css_entries`/`profiles` already are.
- CRUD methods on `Settings` (create/update/delete/list), mirroring
`import_css`/`remove_css`'s pattern: mutate in memory, then `self.save(...)`.
- HTTP endpoints in `src/api.rs` for CRUD, matching the existing settings
endpoints' conventions (routes, status codes, error JSON shape).
- Id generation: reuse the approach in `generate_imported_css_id` (hash
name+content+timestamp) rather than adding a `uuid` dependency.
## Acceptance criteria
- Alert rules round-trip through `save`/`load` like the rest of `Settings`
(see `round_trips_through_save_and_load` for the existing pattern to extend).
- CRUD endpoints create, list, update, and delete rules, and reject unknown
ids the same way `CssError::NotFound` does today.
- No evaluation or delivery logic lives here — this ticket only stores rules.
## Depends on
- [008 — Settings model](008-settings-model.md)
## Depended on by
- [018 — Alert trigger evaluation](018-alert-trigger-evaluation.md)
- [019 — Ntfy sink delivery](019-ntfy-sink-delivery.md)
- [020 — Alerts webapp UI](020-alerts-webapp-ui.md)
-56
View File
@@ -1,56 +0,0 @@
# 018 — Alert trigger evaluation
## Summary
Evaluate `AlertRule` conditions ([017](017-ntfy-alerts.md)) against the live log
stream and fire a delivery ([019](019-ntfy-sink-delivery.md)) when one matches,
without re-scanning already-ingested logs — hook into the same ingest path
`engine.rs` already uses to update timelines/metrics/dashboards on each new
log.
## Scope
Condition types to support, all built on primitives ottyl already has (the
FilterBar query language, [005](005-filterbar-pest-parser.md), and metric
declarations, [002](002-metric-declarations.md)) rather than a new syntax:
- **Immediate** — fire once per log line matching a FilterBar query, as soon
as it's ingested.
- **Threshold** — a declared metric's value, or a FilterBar query's match
count, crosses a fixed level within a rolling time window.
- **New value** — a specified log field takes on a value that hasn't appeared
before (within the alert's lifetime, or a bounded lookback — decide and
document which).
- **Ratio** — the match counts of two FilterBar queries, compared over the
same rolling window, cross a configured ratio.
Each condition stores its own window/threshold parameters (already sketched as
an opaque value on `AlertRule` in 017 — this ticket defines and parses that
shape).
- Evaluation runs inline on ingest (or on bucket update, whichever the engine
already does per-log) — no polling loop and no periodic full re-scan.
- Rolling-window state (counts, last-seen values) is kept in memory per rule;
it does not need to survive a restart.
- On a match, hand off to the configured sink ([019](019-ntfy-sink-delivery.md))
— this ticket owns *deciding* to fire, not *delivering*.
- Respect `AlertRule.enabled`: disabled rules are skipped entirely.
## Acceptance criteria
- Each of the four condition types fires exactly when its condition is met,
and not otherwise (covered by unit tests per type, feeding synthetic logs
through the same ingest path).
- Evaluation cost is proportional to new logs ingested, not to total log
volume — no scan of the full log history per evaluation.
- A disabled rule never fires.
## Depends on
- [017 — Alert definitions & settings persistence](017-ntfy-alerts.md)
- [005 — FilterBar pest parser](005-filterbar-pest-parser.md)
- [002 — Metric declarations](002-metric-declarations.md)
## Depended on by
- [019 — Ntfy sink delivery](019-ntfy-sink-delivery.md)
-44
View File
@@ -1,44 +0,0 @@
# 019 — Ntfy sink delivery
## Summary
Deliver a fired alert ([018](018-alert-trigger-evaluation.md)) to the
[ntfy](https://ntfy.sh) topic configured on its `AlertRule` ([017](017-ntfy-alerts.md)):
a plain HTTP POST of the notification body to `{server}/{topic}`, no SDK
needed.
## Scope
- A small async sender: `POST {server}/{topic}` with the alert's message as
the request body (ntfy's plain-text publish API — no auth, no extra
dependency beyond an HTTP client ottyl can already reach for, or a minimal
one if none exists yet).
- Message content: rule name plus enough context to be useful (matched log
line for Immediate/New value, current value vs. threshold for
Threshold/Ratio).
- Delivery runs off the ingest hot path — spawn it (e.g.
`tokio::spawn`/an mpsc queue) so a slow or unreachable ntfy server never
blocks log ingestion.
- Failure handling: log the error to stderr and drop the notification — no
retry queue or backoff for this ticket (matches the project's existing
"best-effort, log and move on" style, e.g. `remove_css`'s file-removal
fallback).
## Acceptance criteria
- A fired alert results in exactly one HTTP POST to the rule's configured
`{server}/{topic}`.
- Ntfy being unreachable/erroring is logged but does not panic, block
ingestion, or crash the process.
- Covered by a test against a local mock HTTP server asserting the request
path/body.
## Depends on
- [017 — Alert definitions & settings persistence](017-ntfy-alerts.md)
- [018 — Alert trigger evaluation](018-alert-trigger-evaluation.md)
## Depended on by
- [020 — Alerts webapp UI](020-alerts-webapp-ui.md) (for a "send test
notification" action)