phughk/ottyl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Add alert rules with FilterBar-based triggers and ntfy delivery (tickets 017-019)

Browse Source 
Adds AlertRule/AlertCondition/AlertSink models with CRUD persisted on
Settings (mirroring CssEntry's shape), per-line trigger evaluation hooked
into the engine's ingest path (Immediate/Threshold/NewValue/Ratio
conditions), and best-effort ntfy delivery spawned off the ingest hot path.

Design notes:
- Alert queries reuse FilterBar syntax but only support the
  regex()/sequence/timestamp subset (no metric[param] comparisons), since
  resolving those would require a full bucket rescan per alert per ingested
  line rather than cost proportional to new logs.
- Ratio's zero-denominator case skips evaluation for that line rather than
  treating the ratio as infinite.
- NewValue's seen-value set is in-memory only and does not survive a
  restart, matching the engine's other non-persisted runtime state.

engine::run now takes a SettingsHandle (moved from api.rs to settings.rs so
engine.rs can depend on it without a circular import) to read the current
alert rules on each ingest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XFUNZwCvz3kcod3352ZR2o
This commit is contained in:
Hugh Kaznowski
2026-09-05 16:30:14 +01:00
co-authored by Claude Sonnet 5
parent 758feca780
commit dd3320812c
13 changed files with 1579 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+693 -10
View File
@@ -56,7 +56,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [ dependencies = [
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -67,7 +67,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [ dependencies = [
"anstyle", "anstyle",
"once_cell_polyfill", "once_cell_polyfill",
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -134,6 +134,12 @@ dependencies = [
"tracing", "tracing",
] ]
[[package]]
name = "base64"
version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
[[package]] [[package]]
name = "bitflags" name = "bitflags"
version = "2.13.1" version = "2.13.1"
@@ -168,6 +174,23 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
[[package]]
name = "chacha20"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
dependencies = [
"cfg-if",
"cpufeatures",
"rand_core",
]
[[package]] [[package]]
name = "chrono" name = "chrono"
version = "0.4.45" version = "0.4.45"
@@ -233,6 +256,15 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "cpufeatures"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"
dependencies = [
"libc",
]
[[package]] [[package]]
name = "directories" name = "directories"
version = "6.0.0" version = "6.0.0"
@@ -251,7 +283,18 @@ dependencies = [
"libc", "libc",
"option-ext", "option-ext",
"redox_users", "redox_users",
"windows-sys", "windows-sys 0.61.2",
]
[[package]]
name = "displaydoc"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
] ]
[[package]] [[package]]
@@ -261,7 +304,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [ dependencies = [
"libc", "libc",
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -358,8 +401,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"js-sys",
"libc", "libc",
"wasi", "wasi",
"wasm-bindgen",
] ]
[[package]] [[package]]
@@ -369,8 +414,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"js-sys",
"libc", "libc",
"r-efi", "r-efi",
"rand_core",
"wasm-bindgen",
] ]
[[package]] [[package]]
@@ -454,6 +502,23 @@ dependencies = [
"pin-project-lite", "pin-project-lite",
"smallvec", "smallvec",
"tokio", "tokio",
"want",
]
[[package]]
name = "hyper-rustls"
version = "0.27.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
dependencies = [
"http",
"hyper",
"hyper-util",
"rustls",
"tokio",
"tokio-rustls",
"tower-service",
"webpki-roots",
] ]
[[package]] [[package]]
@@ -462,13 +527,21 @@ version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
dependencies = [ dependencies = [
"base64",
"bytes", "bytes",
"futures-channel",
"futures-util",
"http", "http",
"http-body", "http-body",
"hyper", "hyper",
"ipnet",
"libc",
"percent-encoding",
"pin-project-lite", "pin-project-lite",
"socket2",
"tokio", "tokio",
"tower-service", "tower-service",
"tracing",
] ]
[[package]] [[package]]
@@ -495,6 +568,110 @@ dependencies = [
"cc", "cc",
] ]
[[package]]
name = "icu_collections"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513"
dependencies = [
"displaydoc",
"potential_utf",
"utf8_iter",
"yoke",
"zerofrom",
"zerovec",
]
[[package]]
name = "icu_locale_core"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb"
dependencies = [
"displaydoc",
"litemap",
"tinystr",
"writeable",
"zerovec",
]
[[package]]
name = "icu_normalizer"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f"
dependencies = [
"icu_collections",
"icu_normalizer_data",
"icu_properties",
"icu_provider",
"smallvec",
"zerovec",
]
[[package]]
name = "icu_normalizer_data"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0"
[[package]]
name = "icu_properties"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148"
dependencies = [
"displaydoc",
"icu_collections",
"icu_locale_core",
"icu_properties_data",
"icu_provider",
"zerotrie",
"zerovec",
]
[[package]]
name = "icu_properties_data"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa"
[[package]]
name = "icu_provider"
version = "2.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d27bbb9d3abbefac45d55f647c9de1d44aafcd1186eb91879afef17c396c3e73"
dependencies = [
"displaydoc",
"icu_locale_core",
"writeable",
"yoke",
"zerofrom",
"zerotrie",
"zerovec",
]
[[package]]
name = "idna"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de"
dependencies = [
"idna_adapter",
"smallvec",
"utf8_iter",
]
[[package]]
name = "idna_adapter"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714"
dependencies = [
"icu_normalizer",
"icu_properties",
]
[[package]] [[package]]
name = "include_dir" name = "include_dir"
version = "0.7.4" version = "0.7.4"
@@ -514,6 +691,12 @@ dependencies = [
"quote", "quote",
] ]
[[package]]
name = "ipnet"
version = "2.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78"
[[package]] [[package]]
name = "is_terminal_polyfill" name = "is_terminal_polyfill"
version = "1.70.2" version = "1.70.2"
@@ -558,6 +741,12 @@ version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "litemap"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae"
[[package]] [[package]]
name = "lock_api" name = "lock_api"
version = "0.4.14" version = "0.4.14"
@@ -573,6 +762,12 @@ version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]] [[package]]
name = "matchit" name = "matchit"
version = "0.8.4" version = "0.8.4"
@@ -609,7 +804,7 @@ checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8"
dependencies = [ dependencies = [
"libc", "libc",
"wasi", "wasi",
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -654,6 +849,7 @@ dependencies = [
"pest", "pest",
"pest_derive", "pest_derive",
"regex", "regex",
"reqwest",
"serde", "serde",
"serde_json", "serde_json",
"tempfile", "tempfile",
@@ -757,6 +953,15 @@ version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
[[package]]
name = "potential_utf"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661"
dependencies = [
"zerovec",
]
[[package]] [[package]]
name = "proc-macro2" name = "proc-macro2"
version = "1.0.107" version = "1.0.107"
@@ -766,6 +971,62 @@ dependencies = [
"unicode-ident", "unicode-ident",
] ]
[[package]]
name = "quinn"
version = "0.11.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
"rustc-hash",
"rustls",
"socket2",
"thiserror",
"tokio",
"tracing",
"web-time",
]
[[package]]
name = "quinn-proto"
version = "0.11.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "04759210543be93709136e28212294a659ef5001836ff4eab4d663e4529bba83"
dependencies = [
"bytes",
"getrandom 0.4.3",
"lru-slab",
"rand",
"rand_pcg",
"ring",
"rustc-hash",
"rustls",
"rustls-pki-types",
"slab",
"thiserror",
"tinyvec",
"tracing",
"web-time",
]
[[package]]
name = "quinn-udp"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2",
"tracing",
"windows-sys 0.61.2",
]
[[package]] [[package]]
name = "quote" name = "quote"
version = "1.0.47" version = "1.0.47"
@@ -781,6 +1042,32 @@ version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "rand"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
dependencies = [
"chacha20",
"getrandom 0.4.3",
"rand_core",
]
[[package]]
name = "rand_core"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
[[package]]
name = "rand_pcg"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
dependencies = [
"rand_core",
]
[[package]] [[package]]
name = "redox_syscall" name = "redox_syscall"
version = "0.5.18" version = "0.5.18"
@@ -830,6 +1117,64 @@ version = "0.8.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
[[package]]
name = "reqwest"
version = "0.12.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"bytes",
"futures-core",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-rustls",
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
"webpki-roots",
]
[[package]]
name = "ring"
version = "0.17.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
"cfg-if",
"getrandom 0.2.17",
"libc",
"untrusted",
"windows-sys 0.52.0",
]
[[package]]
name = "rustc-hash"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
[[package]] [[package]]
name = "rustix" name = "rustix"
version = "1.1.4" version = "1.1.4"
@@ -840,7 +1185,42 @@ dependencies = [
"errno", "errno",
"libc", "libc",
"linux-raw-sys", "linux-raw-sys",
"windows-sys", "windows-sys 0.61.2",
]
[[package]]
name = "rustls"
version = "0.23.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06"
dependencies = [
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
[[package]]
name = "rustls-pki-types"
version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
dependencies = [
"web-time",
"zeroize",
]
[[package]]
name = "rustls-webpki"
version = "0.103.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
dependencies = [
"ring",
"rustls-pki-types",
"untrusted",
] ]
[[package]] [[package]]
@@ -977,15 +1357,27 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4"
dependencies = [ dependencies = [
"libc", "libc",
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]]
name = "stable_deref_trait"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
[[package]] [[package]]
name = "strsim" name = "strsim"
version = "0.11.1" version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
[[package]]
name = "subtle"
version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]] [[package]]
name = "syn" name = "syn"
version = "2.0.119" version = "2.0.119"
@@ -1013,6 +1405,20 @@ name = "sync_wrapper"
version = "1.0.2" version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
dependencies = [
"futures-core",
]
[[package]]
name = "synstructure"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]] [[package]]
name = "tempfile" name = "tempfile"
@@ -1024,7 +1430,7 @@ dependencies = [
"getrandom 0.4.3", "getrandom 0.4.3",
"once_cell", "once_cell",
"rustix", "rustix",
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -1047,6 +1453,31 @@ dependencies = [
"syn 3.0.5", "syn 3.0.5",
] ]
[[package]]
name = "tinystr"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643"
dependencies = [
"displaydoc",
"zerovec",
]
[[package]]
name = "tinyvec"
version = "1.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cf0ded5c4e56918d8f8a339e1bb67d038d3bc6d144ac407904015ba2e4cde9b"
dependencies = [
"tinyvec_macros",
]
[[package]]
name = "tinyvec_macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
[[package]] [[package]]
name = "tokio" name = "tokio"
version = "1.53.1" version = "1.53.1"
@@ -1061,7 +1492,7 @@ dependencies = [
"signal-hook-registry", "signal-hook-registry",
"socket2", "socket2",
"tokio-macros", "tokio-macros",
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -1075,6 +1506,16 @@ dependencies = [
"syn 3.0.5", "syn 3.0.5",
] ]
[[package]]
name = "tokio-rustls"
version = "0.26.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
dependencies = [
"rustls",
"tokio",
]
[[package]] [[package]]
name = "tokio-util" name = "tokio-util"
version = "0.7.19" version = "0.7.19"
@@ -1125,8 +1566,10 @@ dependencies = [
"pin-project-lite", "pin-project-lite",
"tokio", "tokio",
"tokio-util", "tokio-util",
"tower",
"tower-layer", "tower-layer",
"tower-service", "tower-service",
"url",
] ]
[[package]] [[package]]
@@ -1161,6 +1604,12 @@ dependencies = [
"once_cell", "once_cell",
] ]
[[package]]
name = "try-lock"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
[[package]] [[package]]
name = "ucd-trie" name = "ucd-trie"
version = "0.1.7" version = "0.1.7"
@@ -1179,6 +1628,30 @@ version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "untrusted"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
[[package]]
name = "url"
version = "2.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed"
dependencies = [
"form_urlencoded",
"idna",
"percent-encoding",
"serde",
]
[[package]]
name = "utf8_iter"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
[[package]] [[package]]
name = "utf8parse" name = "utf8parse"
version = "0.2.2" version = "0.2.2"
@@ -1195,6 +1668,15 @@ dependencies = [
"winapi-util", "winapi-util",
] ]
[[package]]
name = "want"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
dependencies = [
"try-lock",
]
[[package]] [[package]]
name = "wasi" name = "wasi"
version = "0.11.1+wasi-snapshot-preview1" version = "0.11.1+wasi-snapshot-preview1"
@@ -1214,6 +1696,16 @@ dependencies = [
"wasm-bindgen-shared", "wasm-bindgen-shared",
] ]
[[package]]
name = "wasm-bindgen-futures"
version = "0.4.78"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ef4c5d3d2cdf5c54f4231181768f5510842e350db025faf1f7163b1030ed928"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]] [[package]]
name = "wasm-bindgen-macro" name = "wasm-bindgen-macro"
version = "0.2.128" version = "0.2.128"
@@ -1246,13 +1738,42 @@ dependencies = [
"unicode-ident", "unicode-ident",
] ]
[[package]]
name = "web-sys"
version = "0.3.105"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9fbddc4a036f00ec4f18c83445bd3115cb306a91da554919a099d9222fe4a7f8"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "web-time"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "webpki-roots"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7dcd9d09a39985f5344844e66b0c530a33843579125f23e21e9f0f220850f22a"
dependencies = [
"rustls-pki-types",
]
[[package]] [[package]]
name = "winapi-util" name = "winapi-util"
version = "0.1.11" version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [ dependencies = [
"windows-sys", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -1314,6 +1835,15 @@ dependencies = [
"windows-link", "windows-link",
] ]
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
]
[[package]] [[package]]
name = "windows-sys" name = "windows-sys"
version = "0.61.2" version = "0.61.2"
@@ -1323,6 +1853,159 @@ dependencies = [
"windows-link", "windows-link",
] ]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "writeable"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc"
[[package]]
name = "yoke"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5"
dependencies = [
"stable_deref_trait",
"yoke-derive",
"zerofrom",
]
[[package]]
name = "yoke-derive"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"synstructure",
]
[[package]]
name = "zerofrom"
version = "0.1.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272"
dependencies = [
"zerofrom-derive",
]
[[package]]
name = "zerofrom-derive"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"synstructure",
]
[[package]]
name = "zeroize"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
[[package]]
name = "zerotrie"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f"
dependencies = [
"displaydoc",
"yoke",
"zerofrom",
]
[[package]]
name = "zerovec"
version = "0.11.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb0464e17806c1d976d5cba29399c7f08e516e279e2ba493f63123b5fca67dd8"
dependencies = [
"yoke",
"zerofrom",
"zerovec-derive",
]
[[package]]
name = "zerovec-derive"
version = "0.11.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
]
[[package]] [[package]]
name = "zmij" name = "zmij"
version = "1.0.23" version = "1.0.23"
Cargo.toml
+1
View File
@@ -18,6 +18,7 @@ humantime = "2"
directories = "6.0.0" directories = "6.0.0"
fs-embed = "0.2.4" fs-embed = "0.2.4"
include_dir = "0.7.4" include_dir = "0.7.4"
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }
[dev-dependencies] [dev-dependencies]
tempfile = "3.27.0" tempfile = "3.27.0"
src/alerts.rs
+654
View File
@@ -0,0 +1,654 @@
//! Alert rules: definitions (ticket 017), trigger evaluation (ticket 018),
//! and ntfy delivery (ticket 019).
//!
//! ## Design: why alert queries only support regex/sequence/timestamp
//!
//! `Immediate`/`Threshold`/`Ratio` conditions reuse FilterBar query strings
//! (see `crate::filter_parser`), evaluated against a *single newly-ingested
//! line* on `EngineMessage::Ingest` (see `evaluate_alerts`, called from
//! `engine::run`). `engine.rs`'s own `compile_filter_query` resolves
//! `metric[param]` comparisons by scanning that metric's *entire* bucket
//! history up front (`compile_metric_comparison`) — reusing that per alert
//! check, per ingested line, would make alert evaluation cost proportional to
//! total log volume rather than to new logs ingested, which this ticket
//! requires. So alert queries here only support the subset of FilterBar that
//! evaluates against a single line's own fields — `regex(...)`, `sequence`,
//! `timestamp`, and `and`/`or` — via `eval_alert_expr` below. A `metric[param]`
//! comparison in an alert query is rejected with a clear error (logged, the
//! rule just never fires) rather than silently misbehaving.
use std::collections::hash_map::DefaultHasher;
use std::collections::{HashMap, HashSet, VecDeque};
use std::fmt;
use std::hash::{Hash, Hasher};
use std::io;
use std::path::Path;
use std::time::{SystemTime, UNIX_EPOCH};
use regex::Regex;
use serde::{Deserialize, Serialize};
use crate::engine::apply_compare_op;
use crate::filter_datetime;
use crate::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::settings::Settings;
use crate::timestamp::Timestamp;
/// A single alert rule definition (ticket 017).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct AlertRule {
pub id: String,
pub name: String,
pub enabled: bool,
pub condition: AlertCondition,
pub sink: AlertSink,
}
/// What triggers an alert rule (ticket 018).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum AlertCondition {
/// Fires once per log line matching `query`.
Immediate { query: String },
/// Fires when `query`'s match count within a rolling `window_ms` crosses
/// `threshold` (via `op`).
Threshold {
query: String,
op: CompareOp,
threshold: f64,
window_ms: i64,
},
/// Fires the first time `field_regex` matches a line and `capture_group`
/// captures a value not seen before *for this rule*. Seen values live
/// only in memory (`AlertRuntimeState::seen_values`) for the process's
/// lifetime — a restart forgets them, so previously-seen values can fire
/// again. This is a deliberate simplicity tradeoff (matches the
/// codebase's general "no restart persistence for in-memory engine
/// state" pattern), not an oversight.
NewValue {
field_regex: String,
capture_group: String,
},
/// Fires when `count(query_a) / count(query_b)` (rolling over
/// `window_ms`) crosses `ratio` (via `op`). A zero `count_b` skips
/// evaluation entirely for that line (rather than treating the ratio as
/// infinite) — there's no meaningful "actual vs threshold" comparison to
/// report yet.
Ratio {
query_a: String,
query_b: String,
op: CompareOp,
ratio: f64,
window_ms: i64,
},
}
/// Where a fired alert is delivered (ticket 019). Modeled as an enum (rather
/// than `AlertRule` holding a bare topic/server) so more sink kinds can be
/// added later without reshaping `AlertRule`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum AlertSink {
Ntfy {
topic: String,
/// Defaults to `https://ntfy.sh` when absent.
server: Option<String>,
},
}
/// Error returned by alert-rule mutations/lookups. Mirrors `CssError`'s shape.
#[derive(Debug)]
pub enum AlertError {
NotFound,
Io(io::Error),
}
impl fmt::Display for AlertError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
AlertError::NotFound => write!(f, "no alert rule with that id"),
AlertError::Io(err) => write!(f, "I/O error: {err}"),
}
}
}
impl std::error::Error for AlertError {}
impl From<io::Error> for AlertError {
fn from(err: io::Error) -> Self {
AlertError::Io(err)
}
}
/// Generates an id for a newly-created alert rule. Same approach as
/// `settings::generate_imported_css_id`: hash the name and current time
/// rather than pulling in a `uuid` dependency.
fn generate_alert_rule_id(name: &str) -> String {
let mut hasher = DefaultHasher::new();
name.hash(&mut hasher);
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
nanos.hash(&mut hasher);
format!("alert-{:016x}", hasher.finish())
}
impl Settings {
/// Creates a new alert rule, persisting settings to `settings_path`.
pub fn create_alert_rule(
&mut self,
settings_path: &Path,
name: String,
enabled: bool,
condition: AlertCondition,
sink: AlertSink,
) -> Result<AlertRule, AlertError> {
let rule = AlertRule { id: generate_alert_rule_id(&name), name, enabled, condition, sink };
self.alert_rules.push(rule.clone());
self.save(settings_path)?;
Ok(rule)
}
/// Replaces an existing alert rule's fields (id stays fixed), persisting
/// settings to `settings_path`. Returns `AlertError::NotFound` if no rule
/// with `id` exists.
pub fn update_alert_rule(
&mut self,
settings_path: &Path,
id: &str,
name: String,
enabled: bool,
condition: AlertCondition,
sink: AlertSink,
) -> Result<AlertRule, AlertError> {
let rule = self
.alert_rules
.iter_mut()
.find(|rule| rule.id == id)
.ok_or(AlertError::NotFound)?;
rule.name = name;
rule.enabled = enabled;
rule.condition = condition;
rule.sink = sink;
let updated = rule.clone();
self.save(settings_path)?;
Ok(updated)
}
/// Removes an alert rule by id, persisting settings to `settings_path`.
/// Returns `AlertError::NotFound` if no rule with `id` exists.
pub fn remove_alert_rule(&mut self, settings_path: &Path, id: &str) -> Result<(), AlertError> {
let index = self
.alert_rules
.iter()
.position(|rule| rule.id == id)
.ok_or(AlertError::NotFound)?;
self.alert_rules.remove(index);
self.save(settings_path)?;
Ok(())
}
}
/// Per-rule in-memory evaluation state (ticket 018): rolling match-timestamp
/// windows for `Threshold`/`Ratio`, and seen-values for `NewValue`. Lives
/// only for the engine process's lifetime (see `AlertCondition::NewValue`'s
/// doc comment) — not persisted, not backfilled against already-ingested
/// logs.
#[derive(Debug, Default)]
pub struct AlertRuntimeState {
threshold_matches: VecDeque<i64>,
ratio_a_matches: VecDeque<i64>,
ratio_b_matches: VecDeque<i64>,
seen_values: HashSet<String>,
}
/// Drops every entry older than `window_ms` relative to `now_ms` from the
/// front of a rolling window. `timestamps` is push_back-ordered, so once a
/// front entry is within the window every entry behind it is too.
fn prune_window(timestamps: &mut VecDeque<i64>, now_ms: i64, window_ms: i64) {
while let Some(&front) = timestamps.front() {
if now_ms - front > window_ms {
timestamps.pop_front();
} else {
break;
}
}
}
/// Evaluates a FilterBar query string against a single log line's own fields
/// only — see this module's doc comment for why `metric[param]` comparisons
/// aren't supported here.
fn eval_alert_query(query: &str, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
let expr = filter_parser::parse(query).map_err(|err| err.to_string())?;
eval_alert_expr(&expr, sequence, timestamp, text)
}
fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
match expr {
FilterExpr::Regex(pattern) => {
let re = Regex::new(pattern).map_err(|err| format!("invalid regex in alert query: {err}"))?;
Ok(re.is_match(text))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "sequence" => {
let target = match value {
FilterValue::Number(n) => *n,
FilterValue::Raw(raw) => raw
.parse::<f64>()
.map_err(|_| format!("'sequence' comparisons require a numeric value, got {raw:?}"))?,
};
Ok(apply_compare_op((sequence as f64).total_cmp(&target), *op))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "timestamp" => {
let target_ms = match value {
FilterValue::Number(n) => *n as i64,
FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, Timestamp::now())
.map_err(|err| err.to_string())?
.as_millis(),
};
Ok(apply_compare_op(timestamp.as_millis().cmp(&target_ms), *op))
}
FilterExpr::Comparison { metric, .. } => Err(format!(
"alert queries only support regex()/sequence/timestamp (not metric comparisons like \
'{metric}[param]'): metric-based alert conditions aren't supported yet"
)),
FilterExpr::And(lhs, rhs) => {
Ok(eval_alert_expr(lhs, sequence, timestamp, text)? && eval_alert_expr(rhs, sequence, timestamp, text)?)
}
FilterExpr::Or(lhs, rhs) => {
Ok(eval_alert_expr(lhs, sequence, timestamp, text)? || eval_alert_expr(rhs, sequence, timestamp, text)?)
}
}
}
/// Evaluates every enabled rule's condition against one newly-ingested line,
/// mutating each rule's runtime state (rolling windows / seen-values) along
/// the way. Returns `(rule, message)` for every rule that fired — the caller
/// (`engine::run`) hands each off to `deliver_for_rule`. A malformed query
/// (bad regex, unsupported metric comparison) is logged to stderr and simply
/// never matches, rather than panicking or propagating.
pub fn evaluate_alerts(
rules: &[AlertRule],
states: &mut HashMap<String, AlertRuntimeState>,
sequence: u64,
timestamp: Timestamp,
text: &str,
) -> Vec<(AlertRule, String)> {
let mut fired = Vec::new();
for rule in rules {
if !rule.enabled {
continue;
}
let state = states.entry(rule.id.clone()).or_default();
if let Some(message) = evaluate_condition(&rule.condition, state, sequence, timestamp, text, &rule.name) {
fired.push((rule.clone(), message));
}
}
fired
}
fn evaluate_condition(
condition: &AlertCondition,
state: &mut AlertRuntimeState,
sequence: u64,
timestamp: Timestamp,
text: &str,
rule_name: &str,
) -> Option<String> {
match condition {
AlertCondition::Immediate { query } => match eval_alert_query(query, sequence, timestamp, text) {
Ok(true) => Some(format!("{rule_name}: matched line: {text}")),
Ok(false) => None,
Err(err) => {
eprintln!("warning: alert '{rule_name}' query error: {err}");
None
}
},
AlertCondition::Threshold { query, op, threshold, window_ms } => {
match eval_alert_query(query, sequence, timestamp, text) {
Ok(true) => {
let now_ms = timestamp.as_millis();
state.threshold_matches.push_back(now_ms);
prune_window(&mut state.threshold_matches, now_ms, *window_ms);
let count = state.threshold_matches.len() as f64;
apply_compare_op(count.total_cmp(threshold), *op).then(|| {
format!("{rule_name}: match count {count} {op:?} threshold {threshold} (window {window_ms}ms)")
})
}
Ok(false) => None,
Err(err) => {
eprintln!("warning: alert '{rule_name}' query error: {err}");
None
}
}
}
AlertCondition::NewValue { field_regex, capture_group } => {
let re = match Regex::new(field_regex) {
Ok(re) => re,
Err(err) => {
eprintln!("warning: alert '{rule_name}' has an invalid field_regex: {err}");
return None;
}
};
let captures = re.captures(text)?;
let value = captures.name(capture_group)?.as_str().to_string();
state
.seen_values
.insert(value.clone())
.then(|| format!("{rule_name}: new value seen: {value} (line: {text})"))
}
AlertCondition::Ratio { query_a, query_b, op, ratio, window_ms } => {
let matched_a = eval_alert_query(query_a, sequence, timestamp, text);
let matched_b = eval_alert_query(query_b, sequence, timestamp, text);
let (matched_a, matched_b) = match (matched_a, matched_b) {
(Ok(a), Ok(b)) => (a, b),
(Err(err), _) | (_, Err(err)) => {
eprintln!("warning: alert '{rule_name}' query error: {err}");
return None;
}
};
if !matched_a && !matched_b {
return None;
}
let now_ms = timestamp.as_millis();
if matched_a {
state.ratio_a_matches.push_back(now_ms);
prune_window(&mut state.ratio_a_matches, now_ms, *window_ms);
}
if matched_b {
state.ratio_b_matches.push_back(now_ms);
prune_window(&mut state.ratio_b_matches, now_ms, *window_ms);
}
let count_a = state.ratio_a_matches.len() as f64;
let count_b = state.ratio_b_matches.len() as f64;
if count_b == 0.0 {
// Zero denominator: skip evaluation rather than treat the
// ratio as infinite — there's nothing meaningful to compare
// against `ratio` yet.
return None;
}
let observed = count_a / count_b;
apply_compare_op(observed.total_cmp(ratio), *op).then(|| {
format!(
"{rule_name}: ratio {observed:.3} ({count_a}/{count_b}) {op:?} {ratio} (window {window_ms}ms)"
)
})
}
}
}
/// Delivers one fired alert to its rule's configured sink. Errors are logged
/// to stderr and dropped — no retry queue, matching `remove_css`'s
/// best-effort style. Called via `tokio::spawn` from the ingest hot path
/// (`engine::run`) so a slow/unreachable sink never blocks ingestion.
pub async fn deliver_for_rule(rule: AlertRule, message: String) {
match rule.sink {
AlertSink::Ntfy { topic, server } => {
let server = server.unwrap_or_else(|| "https://ntfy.sh".to_string());
if let Err(err) = deliver_ntfy(&server, &topic, &message).await {
eprintln!("warning: failed to deliver alert '{}' via ntfy: {err}", rule.name);
}
}
}
}
/// Publishes `message` to `{server}/{topic}` via ntfy's plain-text publish
/// API (no auth, raw body). Standalone and `AlertRule`-free so a future
/// "send test notification" endpoint can call it directly.
pub async fn deliver_ntfy(server: &str, topic: &str, message: &str) -> Result<(), String> {
let url = format!("{}/{}", server.trim_end_matches('/'), topic);
let response = reqwest::Client::new()
.post(&url)
.body(message.to_string())
.send()
.await
.map_err(|err| err.to_string())?;
if response.status().is_success() {
Ok(())
} else {
Err(format!("ntfy responded with status {}", response.status()))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn line(text: &str, sequence: u64, timestamp_ms: i64) -> (u64, Timestamp, &str) {
(sequence, Timestamp::from_millis(timestamp_ms), text)
}
fn immediate_rule(query: &str) -> AlertRule {
AlertRule {
id: "r1".to_string(),
name: "test rule".to_string(),
enabled: true,
condition: AlertCondition::Immediate { query: query.to_string() },
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
}
}
#[test]
fn alert_rule_crud_round_trips_through_save_and_load() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::load(&settings_path);
let created = settings
.create_alert_rule(
&settings_path,
"spike".to_string(),
true,
AlertCondition::Immediate { query: r#"regex("error")"#.to_string() },
AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
)
.unwrap();
assert_eq!(settings.alert_rules.len(), 1);
let reloaded = Settings::load(&settings_path);
assert_eq!(reloaded.alert_rules, settings.alert_rules);
let updated = settings
.update_alert_rule(
&settings_path,
&created.id,
"renamed".to_string(),
false,
AlertCondition::Immediate { query: r#"regex("warn")"#.to_string() },
AlertSink::Ntfy { topic: "other".to_string(), server: Some("https://example.com".to_string()) },
)
.unwrap();
assert_eq!(updated.id, created.id);
assert_eq!(updated.name, "renamed");
assert!(!updated.enabled);
settings.remove_alert_rule(&settings_path, &created.id).unwrap();
assert!(settings.alert_rules.is_empty());
let reloaded = Settings::load(&settings_path);
assert!(reloaded.alert_rules.is_empty());
}
#[test]
fn alert_rule_mutations_reject_unknown_id() {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::load(&settings_path);
let result = settings.update_alert_rule(
&settings_path,
"does-not-exist",
"x".to_string(),
true,
AlertCondition::Immediate { query: r#"regex("x")"#.to_string() },
AlertSink::Ntfy { topic: "x".to_string(), server: None },
);
assert!(matches!(result, Err(AlertError::NotFound)));
let result = settings.remove_alert_rule(&settings_path, "does-not-exist");
assert!(matches!(result, Err(AlertError::NotFound)));
}
#[test]
fn immediate_fires_on_matching_line_only() {
let rules = vec![immediate_rule(r#"regex("error")"#)];
let mut states = HashMap::new();
let (seq, ts, text) = line("all good", 0, 0);
assert!(evaluate_alerts(&rules, &mut states, seq, ts, text).is_empty());
let (seq, ts, text) = line("an error happened", 1, 1);
let fired = evaluate_alerts(&rules, &mut states, seq, ts, text);
assert_eq!(fired.len(), 1);
assert!(fired[0].1.contains("an error happened"));
}
#[test]
fn disabled_rule_never_fires() {
let mut rule = immediate_rule(r#"regex("error")"#);
rule.enabled = false;
let mut states = HashMap::new();
let (seq, ts, text) = line("error!", 0, 0);
assert!(evaluate_alerts(&[rule], &mut states, seq, ts, text).is_empty());
}
#[test]
fn threshold_fires_once_count_crosses_and_not_before() {
let rule = AlertRule {
id: "r1".to_string(),
name: "spike".to_string(),
enabled: true,
condition: AlertCondition::Threshold {
query: r#"regex("error")"#.to_string(),
op: CompareOp::Ge,
threshold: 3.0,
window_ms: 10_000,
},
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
};
let mut states = HashMap::new();
for (seq, ms) in [(0u64, 0i64), (1, 1000)] {
let fired = evaluate_alerts(&[rule.clone()], &mut states, seq, Timestamp::from_millis(ms), "error");
assert!(fired.is_empty(), "should not fire before threshold is crossed");
}
let fired = evaluate_alerts(&[rule.clone()], &mut states, 2, Timestamp::from_millis(2000), "error");
assert_eq!(fired.len(), 1);
// Stays quiet for a non-matching line even though the window is full.
let fired = evaluate_alerts(&[rule], &mut states, 3, Timestamp::from_millis(2100), "fine");
assert!(fired.is_empty());
}
#[test]
fn threshold_window_expires_old_matches() {
let rule = AlertRule {
id: "r1".to_string(),
name: "spike".to_string(),
enabled: true,
condition: AlertCondition::Threshold {
query: r#"regex("error")"#.to_string(),
op: CompareOp::Ge,
threshold: 2.0,
window_ms: 1_000,
},
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
};
let mut states = HashMap::new();
assert!(evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "error").is_empty());
// Second match arrives well after the first has aged out of the window.
assert!(evaluate_alerts(&[rule], &mut states, 1, Timestamp::from_millis(5_000), "error").is_empty());
}
#[test]
fn new_value_fires_only_on_first_sighting() {
let rule = AlertRule {
id: "r1".to_string(),
name: "new user agent".to_string(),
enabled: true,
condition: AlertCondition::NewValue {
field_regex: r"agent=(?P<agent>\S+)".to_string(),
capture_group: "agent".to_string(),
},
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
};
let mut states = HashMap::new();
let fired = evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "agent=curl");
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule.clone()], &mut states, 1, Timestamp::from_millis(1), "agent=curl");
assert!(fired.is_empty());
let fired = evaluate_alerts(&[rule], &mut states, 2, Timestamp::from_millis(2), "agent=wget");
assert_eq!(fired.len(), 1);
}
#[test]
fn ratio_fires_when_crossing_and_skips_zero_denominator() {
let rule = AlertRule {
id: "r1".to_string(),
name: "error ratio".to_string(),
enabled: true,
condition: AlertCondition::Ratio {
query_a: r#"regex("error")"#.to_string(),
query_b: r#"regex("request")"#.to_string(),
op: CompareOp::Ge,
ratio: 0.5,
window_ms: 10_000,
},
sink: AlertSink::Ntfy { topic: "alerts".to_string(), server: None },
};
let mut states = HashMap::new();
// Only an "error" so far; no "request" yet, so denominator is 0 -> skip.
assert!(evaluate_alerts(&[rule.clone()], &mut states, 0, Timestamp::from_millis(0), "error").is_empty());
// One request: ratio 1/1 = 1.0 >= 0.5 -> fires.
let fired = evaluate_alerts(&[rule.clone()], &mut states, 1, Timestamp::from_millis(1), "request");
assert_eq!(fired.len(), 1);
// A second request: ratio 1/2 = 0.5 still >= 0.5 -> fires again.
let fired = evaluate_alerts(&[rule.clone()], &mut states, 2, Timestamp::from_millis(2), "request");
assert_eq!(fired.len(), 1);
// A third request with no more errors: ratio drops to 1/3 -> no longer fires.
let fired = evaluate_alerts(&[rule], &mut states, 3, Timestamp::from_millis(3), "request");
assert!(fired.is_empty());
}
#[test]
fn metric_comparison_in_alert_query_is_rejected_not_panicking() {
let rule = immediate_rule("some_metric[param]==5");
let mut states = HashMap::new();
// Should log a warning and simply not fire, never panic.
let fired = evaluate_alerts(&[rule], &mut states, 0, Timestamp::from_millis(0), "anything");
assert!(fired.is_empty());
}
#[tokio::test]
async fn deliver_ntfy_posts_message_to_server_topic_path() {
use axum::extract::State;
use axum::routing::post;
use axum::Router;
use std::sync::{Arc, Mutex};
use tokio::net::TcpListener;
#[derive(Clone, Default)]
struct Captured(Arc<Mutex<Option<(String, String)>>>);
async fn capture(
State(captured): State<Captured>,
axum::extract::Path(topic): axum::extract::Path<String>,
body: String,
) -> &'static str {
*captured.0.lock().unwrap() = Some((topic, body));
"ok"
}
let captured = Captured::default();
let app = Router::new().route("/{topic}", post(capture)).with_state(captured.clone());
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
let server = format!("http://{addr}");
deliver_ntfy(&server, "alerts", "hello world").await.unwrap();
let (topic, body) = captured.0.lock().unwrap().clone().unwrap();
assert_eq!(topic, "alerts");
assert_eq!(body, "hello world");
}
}
src/api.rs
+148 -12
View File
@@ -1,33 +1,30 @@
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex}; use std::sync::Arc;
use crate::dto::{ use crate::dto::{
BucketValuesParams, CreateDashboardRequest, CreateProfileRequest, CssEntryDto, DashboardDto, BucketValuesParams, CreateAlertRuleRequest, CreateDashboardRequest, CreateProfileRequest,
ErrorDto, ImportCssRequest, LinkCssRequest, MetricDeclarationDto, ParamDeclarationDto, CssEntryDto, DashboardDto, ErrorDto, ImportCssRequest, LinkCssRequest, MetricDeclarationDto,
LogsParams, ProfileSummaryDto, PromqlParams, SaveDashboardRequest, SettingsDto, ParamDeclarationDto, LogsParams, ProfileSummaryDto, PromqlParams, SaveDashboardRequest,
TimelineParams, UpdateSettingsRequest, SettingsDto, TimelineParams, UpdateAlertRuleRequest, UpdateSettingsRequest,
}; };
use crate::engine::{ use crate::engine::{
DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage, EngineQuery, DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage, EngineQuery,
EngineResponse, EngineResponse,
}; };
use crate::alerts::AlertError;
use crate::profile::{self, Profile, ProfileError, ProfileStore}; use crate::profile::{self, Profile, ProfileError, ProfileStore};
use crate::settings::{CssError, Settings}; use crate::settings::{CssError, SettingsHandle};
use axum::extract::{Path as AxumPath, Query, State}; use axum::extract::{Path as AxumPath, Query, State};
use axum::http::{header, StatusCode}; use axum::http::{header, StatusCode};
use axum::response::{IntoResponse, Json, Response}; use axum::response::{IntoResponse, Json, Response};
use axum::routing::{delete, get, post}; use axum::routing::{delete, get, post, put};
use axum::Router; use axum::Router;
use fs_embed::Dir as EmbedDir; use fs_embed::Dir as EmbedDir;
use tokio::sync::oneshot; use tokio::sync::oneshot;
static WEBAPP_DIST: EmbedDir = fs_embed::fs_embed!("webapp/dist"); static WEBAPP_DIST: EmbedDir = fs_embed::fs_embed!("webapp/dist");
/// Shared, process-wide settings handle. Cheap to clone; lock is only held
/// briefly since settings changes are rare (unlike the engine's hot path).
pub type SettingsHandle = Arc<Mutex<Settings>>;
#[derive(Clone)] #[derive(Clone)]
struct ApiState { struct ApiState {
engine: EngineHandle, engine: EngineHandle,
@@ -108,6 +105,11 @@ pub fn router(engine: EngineHandle, settings: SettingsHandle, settings_path: Pat
.route("/api/settings/css/link", post(link_css)) .route("/api/settings/css/link", post(link_css))
.route("/api/settings/css/{id}", delete(remove_css)) .route("/api/settings/css/{id}", delete(remove_css))
.route("/api/settings/css/{id}/content", get(get_css_content)) .route("/api/settings/css/{id}/content", get(get_css_content))
.route("/api/settings/alerts", get(list_alert_rules).post(create_alert_rule))
.route(
"/api/settings/alerts/{id}",
put(update_alert_rule).delete(delete_alert_rule),
)
.route("/api/settings/profiles", get(list_profiles).post(create_profile)) .route("/api/settings/profiles", get(list_profiles).post(create_profile))
.route("/api/settings/profiles/{name}", delete(delete_profile)) .route("/api/settings/profiles/{name}", delete(delete_profile))
.route("/api/settings/profiles/{name}/activate", post(activate_profile)) .route("/api/settings/profiles/{name}/activate", post(activate_profile))
@@ -464,6 +466,63 @@ async fn get_css_content(
} }
} }
/// Maps an `AlertError` to a response status/message: `NotFound` -> 404,
/// `Io` -> 500. Mirrors `css_error_response` above.
fn alert_error_response(err: AlertError) -> axum::response::Response {
match err {
AlertError::NotFound => (StatusCode::NOT_FOUND, err.to_string()).into_response(),
AlertError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(),
}
}
/// `GET /api/settings/alerts` — lists all configured alert rules as
/// `Vec<AlertRule>`.
async fn list_alert_rules(State(state): State<ApiState>) -> impl IntoResponse {
let settings = state.settings.lock().unwrap();
Json(settings.alert_rules.clone()).into_response()
}
/// `POST /api/settings/alerts` — creates an alert rule. Body:
/// `{ "name": "...", "enabled": true, "condition": { "kind": "immediate" | \
/// "threshold" | "newvalue" | "ratio", ... }, "sink": { "kind": "ntfy", \
/// "topic": "...", "server": "..." | null } }`. Returns `201 Created` with
/// the created rule (including its generated `id`).
async fn create_alert_rule(
State(state): State<ApiState>,
Json(body): Json<CreateAlertRuleRequest>,
) -> impl IntoResponse {
let mut settings = state.settings.lock().unwrap();
match settings.create_alert_rule(&state.settings_path, body.name, body.enabled, body.condition, body.sink) {
Ok(rule) => (StatusCode::CREATED, Json(rule)).into_response(),
Err(err) => alert_error_response(err),
}
}
/// `PUT /api/settings/alerts/:id` — replaces an alert rule's fields (id in
/// the URL is authoritative). Returns the updated rule, or `404 Not Found`
/// if no rule with `id` exists.
async fn update_alert_rule(
State(state): State<ApiState>,
AxumPath(id): AxumPath<String>,
Json(body): Json<UpdateAlertRuleRequest>,
) -> impl IntoResponse {
let mut settings = state.settings.lock().unwrap();
match settings.update_alert_rule(&state.settings_path, &id, body.name, body.enabled, body.condition, body.sink) {
Ok(rule) => Json(rule).into_response(),
Err(err) => alert_error_response(err),
}
}
/// `DELETE /api/settings/alerts/:id` — removes an alert rule. Returns
/// `204 No Content`, or `404 Not Found` if no rule with `id` exists.
async fn delete_alert_rule(State(state): State<ApiState>, AxumPath(id): AxumPath<String>) -> impl IntoResponse {
let mut settings = state.settings.lock().unwrap();
match settings.remove_alert_rule(&state.settings_path, &id) {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(err) => alert_error_response(err),
}
}
/// `GET /api/settings` — returns the subset of `Settings` the NavMenu UI /// `GET /api/settings` — returns the subset of `Settings` the NavMenu UI
/// needs (colour scheme, active profile name) as `SettingsDto`. CSS entries /// needs (colour scheme, active profile name) as `SettingsDto`. CSS entries
/// have their own dedicated list endpoint (`GET /api/settings/css`). /// have their own dedicated list endpoint (`GET /api/settings/css`).
@@ -744,11 +803,13 @@ async fn delete_dashboard(
mod tests { mod tests {
use super::*; use super::*;
use crate::engine::{self, IngestedLine}; use crate::engine::{self, IngestedLine};
use crate::settings::Settings;
use crate::timestamp::Timestamp; use crate::timestamp::Timestamp;
use axum::body::Body; use axum::body::Body;
use axum::http::Request; use axum::http::Request;
use http_body_util::BodyExt; use http_body_util::BodyExt;
use serde_json::{json, Value}; use serde_json::{json, Value};
use std::sync::Mutex;
use tokio::sync::mpsc; use tokio::sync::mpsc;
use tower::ServiceExt; use tower::ServiceExt;
@@ -758,12 +819,13 @@ mod tests {
/// `TempDir` guard (kept alive for the test's duration). /// `TempDir` guard (kept alive for the test's duration).
fn test_app() -> (Router, EngineHandle, SettingsHandle, tempfile::TempDir) { fn test_app() -> (Router, EngineHandle, SettingsHandle, tempfile::TempDir) {
let (engine_tx, engine_rx) = mpsc::channel(128); let (engine_tx, engine_rx) = mpsc::channel(128);
tokio::spawn(engine::run(engine_rx));
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json"); let settings_path = dir.path().join("settings.json");
let settings = Arc::new(Mutex::new(Settings::load(&settings_path))); let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
tokio::spawn(engine::run(engine_rx, settings.clone()));
let app = router(engine_tx.clone(), settings.clone(), settings_path); let app = router(engine_tx.clone(), settings.clone(), settings_path);
(app, engine_tx, settings, dir) (app, engine_tx, settings, dir)
} }
@@ -799,6 +861,80 @@ mod tests {
(status, value) (status, value)
} }
#[tokio::test]
async fn alert_rule_crud_round_trips_through_http() {
let (app, _engine, _settings, _dir) = test_app();
let (status, body) = send_json(
&app,
"POST",
"/api/settings/alerts",
json!({
"name": "error spike",
"enabled": true,
"condition": { "kind": "immediate", "query": "regex(\"error\")" },
"sink": { "kind": "ntfy", "topic": "alerts", "server": null }
}),
)
.await;
assert_eq!(status, StatusCode::CREATED);
assert_eq!(body["name"], "error spike");
let id = body["id"].as_str().unwrap().to_string();
let (status, body) = get_json(&app, "/api/settings/alerts").await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body.as_array().unwrap().len(), 1);
let (status, body) = send_json(
&app,
"PUT",
&format!("/api/settings/alerts/{id}"),
json!({
"name": "renamed",
"enabled": false,
"condition": { "kind": "immediate", "query": "regex(\"warn\")" },
"sink": { "kind": "ntfy", "topic": "alerts", "server": null }
}),
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body["name"], "renamed");
assert_eq!(body["enabled"], false);
let status = delete_request(&app, &format!("/api/settings/alerts/{id}")).await;
assert_eq!(status, StatusCode::NO_CONTENT);
let (_, body) = get_json(&app, "/api/settings/alerts").await;
assert!(body.as_array().unwrap().is_empty());
}
#[tokio::test]
async fn alert_rule_routes_404_for_unknown_id() {
let (app, _engine, _settings, _dir) = test_app();
let status = delete_request(&app, "/api/settings/alerts/does-not-exist").await;
assert_eq!(status, StatusCode::NOT_FOUND);
// `update_alert_rule`'s `AlertError::NotFound` maps to a plain-text
// 404 body (mirrors `css_error_response`), not JSON, so this checks
// the status directly rather than via `send_json`'s JSON parsing.
let request = Request::builder()
.method("PUT")
.uri("/api/settings/alerts/does-not-exist")
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(
json!({
"name": "x",
"enabled": true,
"condition": { "kind": "immediate", "query": "regex(\"x\")" },
"sink": { "kind": "ntfy", "topic": "x", "server": null }
})
.to_string(),
))
.unwrap();
let response = app.clone().oneshot(request).await.unwrap();
assert_eq!(response.status(), StatusCode::NOT_FOUND);
}
#[tokio::test] #[tokio::test]
async fn declares_param_and_returns_201() { async fn declares_param_and_returns_201() {
let (app, _engine, _settings, _dir) = test_app(); let (app, _engine, _settings, _dir) = test_app();
src/dto.rs
+23
View File
@@ -1,3 +1,4 @@
use crate::alerts::{AlertCondition, AlertSink};
use crate::data_value::{MetricPrimitiveKind, RegexToMetricPrimitiveMatcher}; use crate::data_value::{MetricPrimitiveKind, RegexToMetricPrimitiveMatcher};
use crate::metric_primitive::MetricPrimitive; use crate::metric_primitive::MetricPrimitive;
use crate::profile::{ChartType, Dashboard, Panel, Profile}; use crate::profile::{ChartType, Dashboard, Panel, Profile};
@@ -236,6 +237,28 @@ where
Option::<T>::deserialize(deserializer).map(Some) Option::<T>::deserialize(deserializer).map(Some)
} }
/// Request body for `POST /api/settings/alerts` (ticket 017): creates a new
/// alert rule (see `crate::alerts::Settings::create_alert_rule`). `AlertRule`
/// itself is returned directly as the response body (it already derives
/// `Serialize`), so there's no separate response DTO here.
#[derive(Debug, Deserialize)]
pub struct CreateAlertRuleRequest {
pub name: String,
pub enabled: bool,
pub condition: AlertCondition,
pub sink: AlertSink,
}
/// Request body for `PUT /api/settings/alerts/:id`: replaces every field
/// except `id` (taken from the URL, see `Settings::update_alert_rule`).
#[derive(Debug, Deserialize)]
pub struct UpdateAlertRuleRequest {
pub name: String,
pub enabled: bool,
pub condition: AlertCondition,
pub sink: AlertSink,
}
/// Summary of one profile for `GET /api/settings/profiles` (ticket 011): /// Summary of one profile for `GET /api/settings/profiles` (ticket 011):
/// just the name and declaration/dashboard counts the Presets panel needs to /// just the name and declaration/dashboard counts the Presets panel needs to
/// render a list, rather than the full param/metric declaration bodies. /// render a list, rather than the full param/metric declaration bodies.
src/engine.rs
+20 -3
View File
@@ -1,3 +1,4 @@
use crate::alerts::{self, AlertRuntimeState};
use crate::data_value::{DataValue, MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher}; use crate::data_value::{DataValue, MetricPrimitiveKind, ParamRole, RegexToMetricPrimitiveMatcher};
use crate::dto::{ use crate::dto::{
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto, BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
@@ -8,11 +9,12 @@ use crate::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::index_key::IndexKey; use crate::index_key::IndexKey;
use crate::metric_primitive::{MetricPrimitive, Number}; use crate::metric_primitive::{MetricPrimitive, Number};
use crate::promql::{self, AggOp, LabelMatcher, PromqlQuery}; use crate::promql::{self, AggOp, LabelMatcher, PromqlQuery};
use crate::settings::SettingsHandle;
use crate::timestamp::Timestamp; use crate::timestamp::Timestamp;
use regex::Regex; use regex::Regex;
use serde::Deserialize; use serde::Deserialize;
use std::cmp::Ordering; use std::cmp::Ordering;
use std::collections::{BTreeMap, HashSet}; use std::collections::{BTreeMap, HashMap, HashSet};
use tokio::sync::{mpsc, oneshot}; use tokio::sync::{mpsc, oneshot};
pub struct IngestedLine { pub struct IngestedLine {
@@ -157,10 +159,14 @@ pub type EngineHandle = mpsc::Sender<EngineMessage>;
/// Sole owner of the primary index. Runs single-threaded so the map never needs a /// Sole owner of the primary index. Runs single-threaded so the map never needs a
/// lock, and does its own response serialization/DTO construction so callers never /// lock, and does its own response serialization/DTO construction so callers never
/// have to clone the index to answer a query. /// have to clone the index to answer a query.
pub async fn run(mut rx: mpsc::Receiver<EngineMessage>) { pub async fn run(mut rx: mpsc::Receiver<EngineMessage>, settings: SettingsHandle) {
let mut index: BTreeMap<IndexKey, DataValue> = BTreeMap::new(); let mut index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let mut max_logs: Option<usize> = None; let mut max_logs: Option<usize> = None;
let mut log_count: usize = 0; let mut log_count: usize = 0;
// Ticket 018: per-rule rolling windows / seen-value sets, keyed by rule
// id. Lives only as long as this loop (no restart persistence — see
// `alerts::AlertCondition::NewValue`'s doc comment).
let mut alert_states: HashMap<String, AlertRuntimeState> = HashMap::new();
while let Some(message) = rx.recv().await { while let Some(message) = rx.recv().await {
match message { match message {
@@ -173,6 +179,15 @@ pub async fn run(mut rx: mpsc::Receiver<EngineMessage>) {
log_count += 1; log_count += 1;
apply_all_metrics(&mut index, sequence, timestamp, &text); apply_all_metrics(&mut index, sequence, timestamp, &text);
evict_excess_logs(&mut index, &mut log_count, max_logs); evict_excess_logs(&mut index, &mut log_count, max_logs);
// Ticket 018: evaluate alert rules against just this line
// (not the whole log range), then hand any fired rule off to
// ticket 019's delivery, off the hot path via `tokio::spawn`.
let rules = settings.lock().unwrap().alert_rules.clone();
let fired = alerts::evaluate_alerts(&rules, &mut alert_states, sequence, timestamp, &text);
for (rule, message) in fired {
tokio::spawn(alerts::deliver_for_rule(rule, message));
}
} }
EngineMessage::SetMaxLogs(new_max) => { EngineMessage::SetMaxLogs(new_max) => {
max_logs = new_max; max_logs = new_max;
@@ -739,7 +754,9 @@ fn compare_metric_primitive(
Ok(ordering.is_some_and(|ordering| apply_compare_op(ordering, op))) Ok(ordering.is_some_and(|ordering| apply_compare_op(ordering, op)))
} }
fn apply_compare_op(ordering: Ordering, op: CompareOp) -> bool { /// `pub(crate)` so `crate::alerts` can reuse it for `Threshold`/`Ratio`
/// comparisons instead of duplicating the match.
pub(crate) fn apply_compare_op(ordering: Ordering, op: CompareOp) -> bool {
match op { match op {
CompareOp::Lt => ordering == Ordering::Less, CompareOp::Lt => ordering == Ordering::Less,
CompareOp::Gt => ordering == Ordering::Greater, CompareOp::Gt => ordering == Ordering::Greater,
src/filter_parser.rs
+6 -2
View File
@@ -26,6 +26,7 @@
use pest::Parser; use pest::Parser;
use pest::iterators::Pair; use pest::iterators::Pair;
use pest_derive::Parser as PestParser; use pest_derive::Parser as PestParser;
use serde::{Deserialize, Serialize};
#[derive(PestParser)] #[derive(PestParser)]
#[grammar = "filter_parser.pest"] #[grammar = "filter_parser.pest"]
@@ -47,8 +48,11 @@ pub enum FilterExpr {
Or(Box<FilterExpr>, Box<FilterExpr>), Or(Box<FilterExpr>, Box<FilterExpr>),
} }
/// Comparison operators supported by metric comparisons. /// Comparison operators supported by metric comparisons. Also reused by
#[derive(Debug, Clone, Copy, PartialEq, Eq)] /// `crate::alerts::AlertCondition::Threshold`/`Ratio` for their own
/// threshold/ratio comparisons (hence `Serialize`/`Deserialize`).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum CompareOp { pub enum CompareOp {
Lt, Lt,
Gt, Gt,
src/main.rs
+2 -1
View File
@@ -1,3 +1,4 @@
mod alerts;
mod api; mod api;
mod data_value; mod data_value;
mod dto; mod dto;
@@ -53,7 +54,7 @@ async fn main() {
let settings = Arc::new(Mutex::new(Settings::load(&settings_path))); let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
let (engine_tx, engine_rx) = mpsc::channel(1024); let (engine_tx, engine_rx) = mpsc::channel(1024);
tokio::spawn(engine::run(engine_rx)); tokio::spawn(engine::run(engine_rx, settings.clone()));
{ {
let max_logs = settings.lock().unwrap().max_logs; let max_logs = settings.lock().unwrap().max_logs;
let _ = engine_tx.send(engine::EngineMessage::SetMaxLogs(max_logs)).await; let _ = engine_tx.send(engine::EngineMessage::SetMaxLogs(max_logs)).await;
src/profile.rs
+4 -2
View File
@@ -932,7 +932,8 @@ mod tests {
let default_profile = store.get(DEFAULT_PROFILE_NAME).unwrap().clone(); let default_profile = store.get(DEFAULT_PROFILE_NAME).unwrap().clone();
let (engine_tx, engine_rx) = mpsc::channel(128); let (engine_tx, engine_rx) = mpsc::channel(128);
tokio::spawn(engine::run(engine_rx)); let settings_handle = std::sync::Arc::new(std::sync::Mutex::new(settings.clone()));
tokio::spawn(engine::run(engine_rx, settings_handle));
load_into_engine(&engine_tx, &default_profile).await; load_into_engine(&engine_tx, &default_profile).await;
for (sequence, text) in [(0u64, "all good"), (1, "an ERROR happened"), (2, "fine again")] { for (sequence, text) in [(0u64, "all good"), (1, "an ERROR happened"), (2, "fine again")] {
@@ -1130,7 +1131,8 @@ mod tests {
// Spawn a fresh engine and replay the profile's declarations into it. // Spawn a fresh engine and replay the profile's declarations into it.
let (engine_tx, engine_rx) = mpsc::channel(16); let (engine_tx, engine_rx) = mpsc::channel(16);
tokio::spawn(crate::engine::run(engine_rx)); let settings_handle = std::sync::Arc::new(std::sync::Mutex::new(settings.clone()));
tokio::spawn(crate::engine::run(engine_rx, settings_handle));
load_into_engine(&engine_tx, &profile).await; load_into_engine(&engine_tx, &profile).await;
// Ingest a line and confirm the engine is functioning with the replayed // Ingest a line and confirm the engine is functioning with the replayed
src/settings.rs
+28 -1
View File
@@ -11,11 +11,18 @@ use std::fs;
use std::hash::{Hash, Hasher}; use std::hash::{Hash, Hasher};
use std::io; use std::io;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
use directories::ProjectDirs; use directories::ProjectDirs;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use crate::alerts::AlertRule;
/// Shared, process-wide settings handle. Cheap to clone; lock is only held
/// briefly since settings changes are rare (unlike the engine's hot path).
pub type SettingsHandle = Arc<Mutex<Settings>>;
/// User's preferred colour scheme for the webapp. /// User's preferred colour scheme for the webapp.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")] #[serde(rename_all = "lowercase")]
@@ -63,7 +70,10 @@ pub struct Profiles {
} }
/// The single, process-wide settings instance. /// The single, process-wide settings instance.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] ///
/// Only `PartialEq` (not `Eq`): `alert_rules` carries `f64` fields
/// (`AlertCondition::Threshold`/`Ratio`), which aren't `Eq`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Settings { pub struct Settings {
pub color_scheme: ColorScheme, pub color_scheme: ColorScheme,
pub css_entries: Vec<CssEntry>, pub css_entries: Vec<CssEntry>,
@@ -72,6 +82,10 @@ pub struct Settings {
/// means unlimited — matches today's unbounded retention. /// means unlimited — matches today's unbounded retention.
#[serde(default)] #[serde(default)]
pub max_logs: Option<usize>, pub max_logs: Option<usize>,
/// Alert rule definitions (ticket 017). `#[serde(default)]` so existing
/// settings.json files without this field still load.
#[serde(default)]
pub alert_rules: Vec<AlertRule>,
} }
impl Default for Settings { impl Default for Settings {
@@ -85,6 +99,7 @@ impl Default for Settings {
profiles_dir: "profiles".to_string(), profiles_dir: "profiles".to_string(),
}, },
max_logs: None, max_logs: None,
alert_rules: Vec::new(),
} }
} }
} }
@@ -355,6 +370,18 @@ mod tests {
}); });
settings.profiles.names.push("default".to_string()); settings.profiles.names.push("default".to_string());
settings.profiles.active_profile = Some("default".to_string()); settings.profiles.active_profile = Some("default".to_string());
settings.alert_rules.push(crate::alerts::AlertRule {
id: "rule1".to_string(),
name: "error spike".to_string(),
enabled: true,
condition: crate::alerts::AlertCondition::Immediate {
query: r#"regex("error")"#.to_string(),
},
sink: crate::alerts::AlertSink::Ntfy {
topic: "alerts".to_string(),
server: None,
},
});
settings.save(&path).unwrap(); settings.save(&path).unwrap();
tickets/017-ntfy-alerts.md
-49
View File
@@ -1,49 +0,0 @@
# 017 — Alert definitions & settings persistence
## Summary
Foundation for the alerting feature: a persisted `AlertRule` model plus CRUD,
following the shape `Settings` already uses for `CssEntry`/`Profiles`
([008](008-settings-model.md)) — one process-wide list, saved to disk on every
mutation.
An `AlertRule` has:
- A **condition** (see [018](018-alert-trigger-evaluation.md) for the types and
how they're evaluated) — for this ticket, just store it as an opaque, already
parsed/validated value; don't implement evaluation here.
- A **sink**: for now, only `Ntfy { topic: String, server: Option<String> }`
(`server` defaults to `https://ntfy.sh` when absent) — modeled as an enum so
[additional sinks](020-alerts-webapp-ui.md) can be added later without
reshaping `AlertRule`.
- Bookkeeping: `id`, `name`, `enabled: bool`.
## Scope
- `AlertRule` struct (or similar) in a new `src/alerts.rs`, `Vec<AlertRule>`
added to `Settings` (`src/settings.rs`), persisted the same way
`css_entries`/`profiles` already are.
- CRUD methods on `Settings` (create/update/delete/list), mirroring
`import_css`/`remove_css`'s pattern: mutate in memory, then `self.save(...)`.
- HTTP endpoints in `src/api.rs` for CRUD, matching the existing settings
endpoints' conventions (routes, status codes, error JSON shape).
- Id generation: reuse the approach in `generate_imported_css_id` (hash
name+content+timestamp) rather than adding a `uuid` dependency.
## Acceptance criteria
- Alert rules round-trip through `save`/`load` like the rest of `Settings`
(see `round_trips_through_save_and_load` for the existing pattern to extend).
- CRUD endpoints create, list, update, and delete rules, and reject unknown
ids the same way `CssError::NotFound` does today.
- No evaluation or delivery logic lives here — this ticket only stores rules.
## Depends on
- [008 — Settings model](008-settings-model.md)
## Depended on by
- [018 — Alert trigger evaluation](018-alert-trigger-evaluation.md)
- [019 — Ntfy sink delivery](019-ntfy-sink-delivery.md)
- [020 — Alerts webapp UI](020-alerts-webapp-ui.md)
tickets/018-alert-trigger-evaluation.md
-56
View File
@@ -1,56 +0,0 @@
# 018 — Alert trigger evaluation
## Summary
Evaluate `AlertRule` conditions ([017](017-ntfy-alerts.md)) against the live log
stream and fire a delivery ([019](019-ntfy-sink-delivery.md)) when one matches,
without re-scanning already-ingested logs — hook into the same ingest path
`engine.rs` already uses to update timelines/metrics/dashboards on each new
log.
## Scope
Condition types to support, all built on primitives ottyl already has (the
FilterBar query language, [005](005-filterbar-pest-parser.md), and metric
declarations, [002](002-metric-declarations.md)) rather than a new syntax:
- **Immediate** — fire once per log line matching a FilterBar query, as soon
as it's ingested.
- **Threshold** — a declared metric's value, or a FilterBar query's match
count, crosses a fixed level within a rolling time window.
- **New value** — a specified log field takes on a value that hasn't appeared
before (within the alert's lifetime, or a bounded lookback — decide and
document which).
- **Ratio** — the match counts of two FilterBar queries, compared over the
same rolling window, cross a configured ratio.
Each condition stores its own window/threshold parameters (already sketched as
an opaque value on `AlertRule` in 017 — this ticket defines and parses that
shape).
- Evaluation runs inline on ingest (or on bucket update, whichever the engine
already does per-log) — no polling loop and no periodic full re-scan.
- Rolling-window state (counts, last-seen values) is kept in memory per rule;
it does not need to survive a restart.
- On a match, hand off to the configured sink ([019](019-ntfy-sink-delivery.md))
— this ticket owns *deciding* to fire, not *delivering*.
- Respect `AlertRule.enabled`: disabled rules are skipped entirely.
## Acceptance criteria
- Each of the four condition types fires exactly when its condition is met,
and not otherwise (covered by unit tests per type, feeding synthetic logs
through the same ingest path).
- Evaluation cost is proportional to new logs ingested, not to total log
volume — no scan of the full log history per evaluation.
- A disabled rule never fires.
## Depends on
- [017 — Alert definitions & settings persistence](017-ntfy-alerts.md)
- [005 — FilterBar pest parser](005-filterbar-pest-parser.md)
- [002 — Metric declarations](002-metric-declarations.md)
## Depended on by
- [019 — Ntfy sink delivery](019-ntfy-sink-delivery.md)
tickets/019-ntfy-sink-delivery.md
-44
View File
@@ -1,44 +0,0 @@
# 019 — Ntfy sink delivery
## Summary
Deliver a fired alert ([018](018-alert-trigger-evaluation.md)) to the
[ntfy](https://ntfy.sh) topic configured on its `AlertRule` ([017](017-ntfy-alerts.md)):
a plain HTTP POST of the notification body to `{server}/{topic}`, no SDK
needed.
## Scope
- A small async sender: `POST {server}/{topic}` with the alert's message as
the request body (ntfy's plain-text publish API — no auth, no extra
dependency beyond an HTTP client ottyl can already reach for, or a minimal
one if none exists yet).
- Message content: rule name plus enough context to be useful (matched log
line for Immediate/New value, current value vs. threshold for
Threshold/Ratio).
- Delivery runs off the ingest hot path — spawn it (e.g.
`tokio::spawn`/an mpsc queue) so a slow or unreachable ntfy server never
blocks log ingestion.
- Failure handling: log the error to stderr and drop the notification — no
retry queue or backoff for this ticket (matches the project's existing
"best-effort, log and move on" style, e.g. `remove_css`'s file-removal
fallback).
## Acceptance criteria
- A fired alert results in exactly one HTTP POST to the rule's configured
`{server}/{topic}`.
- Ntfy being unreachable/erroring is logged but does not panic, block
ingestion, or crash the process.
- Covered by a test against a local mock HTTP server asserting the request
path/body.
## Depends on
- [017 — Alert definitions & settings persistence](017-ntfy-alerts.md)
- [018 — Alert trigger evaluation](018-alert-trigger-evaluation.md)
## Depended on by
- [020 — Alerts webapp UI](020-alerts-webapp-ui.md) (for a "send test
notification" action)