Add alert rules with FilterBar-based triggers and ntfy delivery (tickets 017-019)
Adds AlertRule/AlertCondition/AlertSink models with CRUD persisted on Settings (mirroring CssEntry's shape), per-line trigger evaluation hooked into the engine's ingest path (Immediate/Threshold/NewValue/Ratio conditions), and best-effort ntfy delivery spawned off the ingest hot path. Design notes: - Alert queries reuse FilterBar syntax but only support the regex()/sequence/timestamp subset (no metric[param] comparisons), since resolving those would require a full bucket rescan per alert per ingested line rather than cost proportional to new logs. - Ratio's zero-denominator case skips evaluation for that line rather than treating the ratio as infinite. - NewValue's seen-value set is in-memory only and does not survive a restart, matching the engine's other non-persisted runtime state. engine::run now takes a SettingsHandle (moved from api.rs to settings.rs so engine.rs can depend on it without a circular import) to read the current alert rules on each ingest. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
758feca780
commit
dd3320812c
@@ -1,49 +0,0 @@
|
||||
# 017 — Alert definitions & settings persistence
|
||||
|
||||
## Summary
|
||||
|
||||
Foundation for the alerting feature: a persisted `AlertRule` model plus CRUD,
|
||||
following the shape `Settings` already uses for `CssEntry`/`Profiles`
|
||||
([008](008-settings-model.md)) — one process-wide list, saved to disk on every
|
||||
mutation.
|
||||
|
||||
An `AlertRule` has:
|
||||
|
||||
- A **condition** (see [018](018-alert-trigger-evaluation.md) for the types and
|
||||
how they're evaluated) — for this ticket, just store it as an opaque, already
|
||||
parsed/validated value; don't implement evaluation here.
|
||||
- A **sink**: for now, only `Ntfy { topic: String, server: Option<String> }`
|
||||
(`server` defaults to `https://ntfy.sh` when absent) — modeled as an enum so
|
||||
[additional sinks](020-alerts-webapp-ui.md) can be added later without
|
||||
reshaping `AlertRule`.
|
||||
- Bookkeeping: `id`, `name`, `enabled: bool`.
|
||||
|
||||
## Scope
|
||||
|
||||
- `AlertRule` struct (or similar) in a new `src/alerts.rs`, `Vec<AlertRule>`
|
||||
added to `Settings` (`src/settings.rs`), persisted the same way
|
||||
`css_entries`/`profiles` already are.
|
||||
- CRUD methods on `Settings` (create/update/delete/list), mirroring
|
||||
`import_css`/`remove_css`'s pattern: mutate in memory, then `self.save(...)`.
|
||||
- HTTP endpoints in `src/api.rs` for CRUD, matching the existing settings
|
||||
endpoints' conventions (routes, status codes, error JSON shape).
|
||||
- Id generation: reuse the approach in `generate_imported_css_id` (hash
|
||||
name+content+timestamp) rather than adding a `uuid` dependency.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Alert rules round-trip through `save`/`load` like the rest of `Settings`
|
||||
(see `round_trips_through_save_and_load` for the existing pattern to extend).
|
||||
- CRUD endpoints create, list, update, and delete rules, and reject unknown
|
||||
ids the same way `CssError::NotFound` does today.
|
||||
- No evaluation or delivery logic lives here — this ticket only stores rules.
|
||||
|
||||
## Depends on
|
||||
|
||||
- [008 — Settings model](008-settings-model.md)
|
||||
|
||||
## Depended on by
|
||||
|
||||
- [018 — Alert trigger evaluation](018-alert-trigger-evaluation.md)
|
||||
- [019 — Ntfy sink delivery](019-ntfy-sink-delivery.md)
|
||||
- [020 — Alerts webapp UI](020-alerts-webapp-ui.md)
|
||||
@@ -1,56 +0,0 @@
|
||||
# 018 — Alert trigger evaluation
|
||||
|
||||
## Summary
|
||||
|
||||
Evaluate `AlertRule` conditions ([017](017-ntfy-alerts.md)) against the live log
|
||||
stream and fire a delivery ([019](019-ntfy-sink-delivery.md)) when one matches,
|
||||
without re-scanning already-ingested logs — hook into the same ingest path
|
||||
`engine.rs` already uses to update timelines/metrics/dashboards on each new
|
||||
log.
|
||||
|
||||
## Scope
|
||||
|
||||
Condition types to support, all built on primitives ottyl already has (the
|
||||
FilterBar query language, [005](005-filterbar-pest-parser.md), and metric
|
||||
declarations, [002](002-metric-declarations.md)) rather than a new syntax:
|
||||
|
||||
- **Immediate** — fire once per log line matching a FilterBar query, as soon
|
||||
as it's ingested.
|
||||
- **Threshold** — a declared metric's value, or a FilterBar query's match
|
||||
count, crosses a fixed level within a rolling time window.
|
||||
- **New value** — a specified log field takes on a value that hasn't appeared
|
||||
before (within the alert's lifetime, or a bounded lookback — decide and
|
||||
document which).
|
||||
- **Ratio** — the match counts of two FilterBar queries, compared over the
|
||||
same rolling window, cross a configured ratio.
|
||||
|
||||
Each condition stores its own window/threshold parameters (already sketched as
|
||||
an opaque value on `AlertRule` in 017 — this ticket defines and parses that
|
||||
shape).
|
||||
|
||||
- Evaluation runs inline on ingest (or on bucket update, whichever the engine
|
||||
already does per-log) — no polling loop and no periodic full re-scan.
|
||||
- Rolling-window state (counts, last-seen values) is kept in memory per rule;
|
||||
it does not need to survive a restart.
|
||||
- On a match, hand off to the configured sink ([019](019-ntfy-sink-delivery.md))
|
||||
— this ticket owns *deciding* to fire, not *delivering*.
|
||||
- Respect `AlertRule.enabled`: disabled rules are skipped entirely.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Each of the four condition types fires exactly when its condition is met,
|
||||
and not otherwise (covered by unit tests per type, feeding synthetic logs
|
||||
through the same ingest path).
|
||||
- Evaluation cost is proportional to new logs ingested, not to total log
|
||||
volume — no scan of the full log history per evaluation.
|
||||
- A disabled rule never fires.
|
||||
|
||||
## Depends on
|
||||
|
||||
- [017 — Alert definitions & settings persistence](017-ntfy-alerts.md)
|
||||
- [005 — FilterBar pest parser](005-filterbar-pest-parser.md)
|
||||
- [002 — Metric declarations](002-metric-declarations.md)
|
||||
|
||||
## Depended on by
|
||||
|
||||
- [019 — Ntfy sink delivery](019-ntfy-sink-delivery.md)
|
||||
@@ -1,44 +0,0 @@
|
||||
# 019 — Ntfy sink delivery
|
||||
|
||||
## Summary
|
||||
|
||||
Deliver a fired alert ([018](018-alert-trigger-evaluation.md)) to the
|
||||
[ntfy](https://ntfy.sh) topic configured on its `AlertRule` ([017](017-ntfy-alerts.md)):
|
||||
a plain HTTP POST of the notification body to `{server}/{topic}`, no SDK
|
||||
needed.
|
||||
|
||||
## Scope
|
||||
|
||||
- A small async sender: `POST {server}/{topic}` with the alert's message as
|
||||
the request body (ntfy's plain-text publish API — no auth, no extra
|
||||
dependency beyond an HTTP client ottyl can already reach for, or a minimal
|
||||
one if none exists yet).
|
||||
- Message content: rule name plus enough context to be useful (matched log
|
||||
line for Immediate/New value, current value vs. threshold for
|
||||
Threshold/Ratio).
|
||||
- Delivery runs off the ingest hot path — spawn it (e.g.
|
||||
`tokio::spawn`/an mpsc queue) so a slow or unreachable ntfy server never
|
||||
blocks log ingestion.
|
||||
- Failure handling: log the error to stderr and drop the notification — no
|
||||
retry queue or backoff for this ticket (matches the project's existing
|
||||
"best-effort, log and move on" style, e.g. `remove_css`'s file-removal
|
||||
fallback).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A fired alert results in exactly one HTTP POST to the rule's configured
|
||||
`{server}/{topic}`.
|
||||
- Ntfy being unreachable/erroring is logged but does not panic, block
|
||||
ingestion, or crash the process.
|
||||
- Covered by a test against a local mock HTTP server asserting the request
|
||||
path/body.
|
||||
|
||||
## Depends on
|
||||
|
||||
- [017 — Alert definitions & settings persistence](017-ntfy-alerts.md)
|
||||
- [018 — Alert trigger evaluation](018-alert-trigger-evaluation.md)
|
||||
|
||||
## Depended on by
|
||||
|
||||
- [020 — Alerts webapp UI](020-alerts-webapp-ui.md) (for a "send test
|
||||
notification" action)
|
||||
Reference in New Issue
Block a user