phughk/ottyl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Compare commits

base: phughk/ottyl:fc8de1a5b02d4cfa0a13854d01ea7ddc270a5ad4
Branches Tags
phughk/ottyl:main
phughk/ottyl:v0.1.0
...
compare: phughk/ottyl:a810ecf396b6dbe3c5941ae1976c9d2d5f85daab
Branches Tags
phughk/ottyl:main
phughk/ottyl:v0.1.0
9 Commits
fc8de1a5b0 ... a810ecf396
Author SHA1 Message Date
Hugh KaznowskiandClaude Sonnet 5 a810ecf396 Split src/ into an api/cli/engine/config module hierarchy 
- src/api/: HTTP router + DTOs + the MCP server (the HTTP/MCP-facing
  surface): mod.rs (formerly api.rs), dto.rs, mcp_server.rs.
- src/cli/: argument parsing and process entry (formerly inline in
  main.rs) plus session registry: mod.rs, sessions.rs. main.rs is now
  a thin entry point that just declares modules and calls cli::run().
- src/engine/: the actor + everything it depends on to parse/index/
  query log lines: mod.rs (formerly engine.rs), alerts.rs, promql.rs,
  rate.rs, ingest.rs, logfmt.rs, filter_parser.rs/.pest,
  filter_datetime.rs, index_key.rs, data_value.rs, metric_primitive.rs,
  timestamp.rs.
- src/config/: persisted configuration: mod.rs (formerly settings.rs),
  profile.rs.
- src/consts.rs stays at the crate root, used from every module above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:15:39 +01:00
Hugh KaznowskiandClaude Sonnet 5 311713986a Wrap the left-right index in IndexWriter/IndexReader types 
The engine actor and API handlers previously called left_right's
WriteHandle/ReadHandleFactory methods (append/publish/enter/handle)
directly. IndexWriter now exposes mutate()/peek() and IndexReader
exposes handle() (returning an IndexReadHandle with its own enter()),
so how the single-writer/multi-reader index is accessed is named and
scoped to these two types instead of being raw left_right calls
scattered through engine.rs/api.rs/profile.rs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:10:23 +01:00
Hugh KaznowskiandClaude Sonnet 5 3c2995acc1 Convert remaining clean-receiver free functions to methods 
Profile (legacy-shape migration, load_into_engine, load_legacy_store),
MetricDeclarationRecord/Dashboard seed constructors, PromqlQuery::parse,
LogStreamHeader (detect/split_row/column_pattern/message_pattern), and
api.rs's EngineHandle::ask/*Error::into_response/ParamDeclarationDto::reserved
all had a single obvious receiver and are now methods on it instead of
free functions taking that type as a parameter.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:08:29 +01:00
Hugh KaznowskiandClaude Sonnet 5 529f79e9da Convert engine.rs index-manipulating functions into EngineSnapshot methods 
Free functions that took the engine's index/sort-helper-index state as
a parameter now live on EngineSnapshot (or, for a handful of clean
single-receiver cases, on MetricPrimitive/CompareOp/CompiledFilter/
EngineQuery). Route handlers in api.rs and alert evaluation in
alerts.rs are updated to call the new methods; alerts.rs now threads
an &EngineSnapshot through instead of a raw &BTreeMap so the per-line
alert backfill path doesn't clone the whole index just to call a method.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 17:03:19 +01:00
Hugh KaznowskiandClaude Sonnet 5 b6401c1c3c Strip all comments outside src/consts.rs 
Removes doc comments, module comments, and inline comments across
the backend so the only comments left in the codebase are the brief
ones on the constants they document.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 16:34:42 +01:00
Hugh KaznowskiandClaude Sonnet 5 284e3562a2 Move scattered magic constants into src/consts.rs 
Centralizes engine tuning values, poll intervals, logfmt column
regex fragments, and app/settings identity strings that were
previously defined ad hoc across engine.rs, ingest.rs, logfmt.rs,
profile.rs, settings.rs, and main.rs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 16:29:03 +01:00
Hugh Kaznowski 1c635bc530 Left-right single-writer, multi-reader index 2026-09-06 16:05:04 +01:00
Hugh KaznowskiandClaude Sonnet 5 9fc7540a5b Resolve tickets 071-074, 076: log-stream contamination, sort, ottyl_fmt column, ingest lag 
- 076: gate auto-declared log-stream column/message params
  (`ParamDeclaration::log_stream_gated`) behind `is_log_stream_line` at
  extraction time, since their fixed-offset regex has no content anchor
  and previously extracted garbage from continuation lines and the
  header row itself (the regex crate doesn't support lookaround, so
  this couldn't be baked into the regex text).
- 074: end-to-end regression test confirms sorting a real `log stream`
  capture by a log-stream column (e.g. PID) now actually reorders
  results, since 076's fix removes the garbage values that were
  corrupting the sort helper index; also add a "Building sort
  index…" indicator in the timeline UI for a slow first-time sort.
- 072: attach the reserved `ottyl_fmt` pseudo-param to each
  `LogEntryDto`'s params so it can be picked as a Log Preview column
  (previously listed as pickable but always blank).
- 073: track a rolling 5s high-watermark of ingestion-to-engine-processing
  lag and surface it in the nav stats popover next to approx_bytes_total.
- 071: group the regex(...) help button back with the filter bar's other
  controls, and restructure the Filtering docs section into scannable
  per-feature subsections (also fixes stale log_type/process/pid mention).

Tickets 070 (CLI scoping) and 075 (API unresponsiveness investigation)
are left in place per their own scope — both are explicitly
conversation/investigation tasks, not implementation tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A1ZJ3SrayzLzVndhVq9BmS
 2026-09-06 15:12:07 +01:00
Hugh Kaznowski f34dbba6c7 Update skill and tickets 2026-09-06 14:37:03 +01:00
37 changed files with 6195 additions and 6778 deletions
Expand all files Collapse all files
Cargo.lock
Generated
+130
View File
@@ -272,6 +272,12 @@ dependencies = [
"libc",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a31eee39dddec8330830986fcd7625edb5a24ec90ea038215273bbc3adb08ac6"
[[package]]
name = "darling"
version = "0.24.1"
@@ -496,6 +502,21 @@ dependencies = [
"slab",
]
[[package]]
name = "generator"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b3b854b0e584ead1a33f18b2fcad7cf7be18b3875c78816b753639aa501513ae"
dependencies = [
"cc",
"cfg-if",
"libc",
"log",
"rustversion",
"windows-link",
"windows-result",
]
[[package]]
name = "getrandom"
version = "0.2.17"
@@ -840,6 +861,23 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "lazy_static"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
[[package]]
name = "left-right"
version = "0.11.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8bc015ded5d9b3054dbbdb63332cdd6ee42352ccef19e911e25117490e2f48ee"
dependencies = [
"crossbeam-utils",
"loom",
"slab",
]
[[package]]
name = "libc"
version = "0.2.189"
@@ -882,12 +920,34 @@ version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
[[package]]
name = "loom"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca"
dependencies = [
"cfg-if",
"generator",
"scoped-tls",
"tracing",
"tracing-subscriber",
]
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "matchers"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
dependencies = [
"regex-automata",
]
[[package]]
name = "matchit"
version = "0.8.4"
@@ -933,6 +993,15 @@ dependencies = [
"minimal-lexical",
]
[[package]]
name = "nu-ansi-term"
version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "num-traits"
version = "0.2.19"
@@ -973,6 +1042,7 @@ dependencies = [
"http-body-util",
"humantime",
"include_dir",
"left-right",
"nom",
"pest",
"pest_derive",
@@ -1481,6 +1551,12 @@ dependencies = [
"syn 3.0.5",
]
[[package]]
name = "scoped-tls"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294"
[[package]]
name = "scopeguard"
version = "1.2.0"
@@ -1564,6 +1640,15 @@ dependencies = [
"serde",
]
[[package]]
name = "sharded-slab"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6"
dependencies = [
"lazy_static",
]
[[package]]
name = "shlex"
version = "2.0.1"
@@ -1701,6 +1786,15 @@ dependencies = [
"syn 3.0.5",
]
[[package]]
name = "thread_local"
version = "1.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070"
dependencies = [
"cfg-if",
]
[[package]]
name = "tinystr"
version = "0.8.4"
@@ -1854,6 +1948,36 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
dependencies = [
"once_cell",
"valuable",
]
[[package]]
name = "tracing-log"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3"
dependencies = [
"log",
"once_cell",
"tracing-core",
]
[[package]]
name = "tracing-subscriber"
version = "0.3.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
dependencies = [
"matchers",
"nu-ansi-term",
"once_cell",
"regex-automata",
"sharded-slab",
"smallvec",
"thread_local",
"tracing",
"tracing-core",
"tracing-log",
]
[[package]]
@@ -1915,6 +2039,12 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "valuable"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
[[package]]
name = "walkdir"
version = "2.5.0"
Cargo.toml
+1
View File
@@ -22,6 +22,7 @@ reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"
petname = { version = "3.2.0", default-features = false, features = ["default-rng", "default-words"] }
rmcp = { version = "3.2.0", default-features = false, features = ["server", "macros", "transport-io"] }
anyhow = "1"
left-right = "0.11.8"
[dev-dependencies]
tempfile = "3.27.0"
SKILL.md
+15
View File
@@ -32,6 +32,21 @@ ottyl mcp --port 51423
This exposes six tools, each a thin proxy onto the target instance's HTTP API.
## Get the latest SKILL.md
This file is compiled into the `ottyl` binary, so a copy you have on disk can lag behind the
version a running instance actually serves (e.g. after an upgrade). A running instance always
serves its own current copy read-only over HTTP — fetch it directly rather than trusting a stale
local file:
```
curl http://localhost:<port>/SKILL.md
```
Use the port from `ottyl sessions` (or the `--port`/`--session` you passed to `ottyl mcp`). There
is no CLI subcommand for this and no way to update it at runtime — the served content only changes
when the binary is rebuilt with a new `SKILL.md`.
## Tools
### `query_logs`
src/dto.rs → src/api/dto.rs
+7 -60
View File
@@ -1,8 +1,8 @@
use crate::alerts::AlertCondition;
use crate::data_value::MetricPrimitiveKind;
use crate::metric_primitive::MetricPrimitive;
use crate::profile::{ChartType, Dashboard, Panel, Profile};
use crate::settings::{ColorScheme, CssEntry, Settings};
use crate::engine::alerts::AlertCondition;
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::metric_primitive::MetricPrimitive;
use crate::config::profile::{ChartType, Dashboard, Panel, Profile};
use crate::config::{ColorScheme, CssEntry, Settings};
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
@@ -11,7 +11,6 @@ pub struct LogEntryDto {
pub sequence: u64,
pub timestamp_ms: i64,
pub text: String,
/// Only params whose regex matched (and parsed as the declared kind) are present — omitted rather than sent as null, so the frontend can render a blank for anything missing.
pub params: BTreeMap<String, MetricPrimitive>,
}
@@ -20,20 +19,10 @@ pub struct ParamDeclarationDto {
pub name: String,
pub message_match_regex: String,
pub capture_group: String,
/// See `crate::data_value::ParamDeclaration::json_field_path`.
pub json_field_path: Option<String>,
pub kind: MetricPrimitiveKind,
/// See `crate::data_value::ParamDeclaration::elapsed_since_last`.
pub elapsed_since_last: bool,
/// See `crate::data_value::ParamDeclaration::auto_derived`. The frontend uses this to disable
/// editing — `POST /api/params` also rejects a request naming an auto-derived param.
pub auto_derived: bool,
/// Set only for the built-in `ottyl_seq`/`ottyl_ts`/`ottyl_fmt` names
/// (`crate::engine::OTTYL_RESERVED_PARAM_NAMES`), synthesized into the `GET /api/params`
/// response by `crate::api::list_params` rather than backed by any index entry — distinct from
/// `auto_derived` (which *is* a real, if machine-discovered, `ParamDeclaration`). Always
/// `false` for every other entry. There is no removal path for these at all (not just a
/// disabled one), so the frontend should treat `reserved` as non-deletable/non-editable.
#[serde(default)]
pub reserved: bool,
}
@@ -42,13 +31,9 @@ pub struct ParamDeclarationDto {
pub struct MetricDeclarationDto {
pub name: String,
pub message_match_regex: String,
/// Names of standalone `ParamDeclaration`s used as this metric's bucket labels.
pub label_params: Vec<String>,
/// Name of the standalone `ParamDeclaration` (if any) whose primitive becomes this metric's per-event bucket value instead of a label.
pub value_param: Option<String>,
/// See `crate::engine::DeclareMetricRequest::matched_labels`.
pub matched_labels: Option<BTreeMap<String, String>>,
/// See `crate::engine::DeclareMetricRequest::unmatched_labels`.
pub unmatched_labels: Option<BTreeMap<String, String>>,
}
@@ -65,19 +50,8 @@ pub struct StatsDto {
pub last_sequence: Option<u64>,
pub first_timestamp_ms: Option<i64>,
pub last_timestamp_ms: Option<i64>,
/// Ticket 011: a running, *approximate* estimate of the in-memory heap
/// held by currently-ingested logs, in bytes. "Approximate" because it
/// only counts each `IngestedLine::text`'s UTF-8 byte length — it does
/// not account for `String`'s allocator overhead/capacity slack, the
/// `BTreeMap`/`IndexKey` node overhead the text is stored under, or any
/// derived data (bucket values, params) the line produces elsewhere in
/// the index. That's judged the best cheap proxy for "how much memory is
/// this log's text costing us": it's O(1) to compute per log (`text.len()`,
/// already an interned byte count for UTF-8 `String`), moves in lockstep
/// with the dominant cost (raw log text, which is usually much larger
/// than the fixed per-entry bookkeeping), and needs no per-entry
/// bookkeeping beyond what eviction already tracks.
pub approx_bytes_total: u64,
pub ingest_lag_high_watermark_ms: i64,
}
#[derive(Debug, Deserialize)]
@@ -86,15 +60,8 @@ pub struct LogsParams {
pub start_ms: Option<i64>,
pub end_ms: Option<i64>,
pub limit: Option<usize>,
/// Raw FilterBar query string, e.g. `regex("error") and ottyl_seq<5`.
pub filter: Option<String>,
/// Ticket 045: an already-declared param name to order the returned entries by, in place of the
/// default ingest order (which tracks `ottyl_ts`). `None` (the default) leaves ordering
/// unchanged.
pub sort_by: Option<String>,
/// Ticket 045: only meaningful alongside `sort_by`. `true` (the default) orders lines missing
/// that param among themselves by `ottyl_seq` and appends them after the sorted lines; `false`
/// drops those lines from the response entirely.
pub sort_missing_fallback: Option<bool>,
}
@@ -103,11 +70,9 @@ pub struct TimelineParams {
pub start_ms: Option<i64>,
pub end_ms: Option<i64>,
pub bucket_ms: Option<i64>,
/// Same FilterBar query string as `LogsParams::filter`; only matching logs are counted into timeline buckets.
pub filter: Option<String>,
}
/// One (sequence, timestamp, value) occurrence within a metric's bucket for a given label combination.
#[derive(Debug, Clone, Serialize)]
pub struct BucketValueDto {
pub labels: BTreeMap<String, MetricPrimitive>,
@@ -116,7 +81,6 @@ pub struct BucketValueDto {
pub value: MetricPrimitive,
}
/// `labels` is a comma-separated `key=value` list (not repeated params) since axum's `Query` extractor needs a fixed set of field names.
#[derive(Debug, Deserialize)]
pub struct BucketValuesParams {
pub labels: Option<String>,
@@ -126,7 +90,6 @@ pub struct BucketValuesParams {
pub limit: Option<usize>,
}
/// `labels` uses plain `String` (already Display-formatted) since charts just need something to show; `value` is always `f64` for plottability.
#[derive(Debug, Clone, Serialize, PartialEq)]
pub struct PromqlSeriesDto {
pub labels: BTreeMap<String, String>,
@@ -140,27 +103,22 @@ pub struct PromqlParams {
pub end_ms: Option<i64>,
}
/// JSON error body returned by API handlers on a 4xx/5xx response.
#[derive(Debug, Serialize)]
pub struct ErrorDto {
pub error: String,
}
/// Every imported CSS entry is a selectable theme (ticket 020) — there's no separate
/// always-on-overlay mode to opt into.
#[derive(Debug, Deserialize)]
pub struct ImportCssRequest {
pub filename: String,
pub content: String,
}
/// A reference to an external CSS path/URL that ottyl doesn't copy or own.
#[derive(Debug, Deserialize)]
pub struct LinkCssRequest {
pub path_or_url: String,
}
/// `id` is the real id for `Imported` entries or a stable id derived from the path/URL for `Linked` ones, so the frontend can key both kinds uniformly (only `Imported` entries are deletable/content-fetchable).
#[derive(Debug, Clone, Serialize)]
pub struct CssEntryDto {
pub id: String,
@@ -186,7 +144,7 @@ impl From<&CssEntry> for CssEntryDto {
path_or_url: None,
},
CssEntry::Linked { path_or_url } => CssEntryDto {
id: crate::settings::css_entry_id(entry),
id: crate::config::css_entry_id(entry),
kind: CssEntryKind::Linked,
filename: None,
path_or_url: Some(path_or_url.clone()),
@@ -195,18 +153,13 @@ impl From<&CssEntry> for CssEntryDto {
}
}
/// `css_entries` is omitted here since it has its own `GET /api/settings/css` endpoint.
#[derive(Debug, Clone, Serialize)]
pub struct SettingsDto {
pub color_scheme: ColorScheme,
pub active_profile: Option<String>,
/// `None` means unlimited.
pub max_logs: Option<usize>,
/// Id of the selected custom theme entry, if any. `None` means `color_scheme` alone determines the active palette.
pub custom_theme_id: Option<String>,
/// Ntfy topic every alert rule delivers to; `None` until configured.
pub ntfy_topic: Option<String>,
/// Ntfy server override; `None` means `https://ntfy.sh`.
pub ntfy_server: Option<String>,
}
@@ -223,7 +176,6 @@ impl From<&Settings> for SettingsDto {
}
}
/// `max_logs`/`custom_theme_id`/`ntfy_topic`/`ntfy_server` distinguish "omitted" (leave unchanged) from "present" (null clears, value sets), via `deserialize_double_option`.
#[derive(Debug, Deserialize)]
pub struct UpdateSettingsRequest {
pub color_scheme: Option<ColorScheme>,
@@ -237,7 +189,6 @@ pub struct UpdateSettingsRequest {
pub ntfy_server: Option<Option<String>>,
}
/// Distinguishes a missing field (outer `None`) from present `null` (`Some(None)`) or present value (`Some(Some(value))`), which plain `Option<Option<T>>` can't do alone.
fn deserialize_double_option<'de, D, T>(deserializer: D) -> Result<Option<Option<T>>, D::Error>
where
D: serde::Deserializer<'de>,
@@ -260,7 +211,6 @@ pub struct UpdateAlertRuleRequest {
pub condition: AlertCondition,
}
/// `server` defaults to `https://ntfy.sh` when absent.
#[derive(Debug, Deserialize)]
pub struct TestAlertNotificationRequest {
pub topic: String,
@@ -291,7 +241,6 @@ pub struct CreateProfileRequest {
pub name: String,
}
/// Mirrors `profile::Panel`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PanelDto {
pub id: String,
@@ -322,7 +271,6 @@ impl From<&PanelDto> for Panel {
}
}
/// Mirrors `profile::Dashboard`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DashboardDto {
pub id: String,
@@ -355,7 +303,6 @@ pub struct CreateDashboardRequest {
pub name: String,
}
/// The dashboard `id` in the URL is authoritative; any `id` in the body is ignored.
#[derive(Debug, Deserialize)]
pub struct SaveDashboardRequest {
pub name: String,
src/mcp_server.rs → src/api/mcp_server.rs
-40
View File
@@ -1,13 +1,3 @@
//! MCP (Model Context Protocol) server for agents (ticket 067): a short-lived process, started
//! via `ottyl mcp --session <name>` (see `crate::sessions`), that speaks MCP over stdio and
//! proxies each tool call to a *running* ottyl instance's existing HTTP API. Every tool here is a
//! thin translation onto `api.rs` routes/DTOs — no query logic is duplicated outside `engine.rs`.
//!
//! Uses the official `rmcp` SDK (https://github.com/modelcontextprotocol/rust-sdk) over the
//! `transport-io` (stdio) transport rather than hand-rolling the JSON-RPC/MCP handshake, since a
//! maintained implementation of the protocol's framing and lifecycle is worth the extra
//! dependency.
use std::collections::BTreeMap;
use rmcp::handler::server::router::tool::ToolRouter;
@@ -21,13 +11,10 @@ use serde_json::Value;
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct QueryLogsParams {
/// FilterBar query string, e.g. `regex("error") and ottyl_seq<5`. Omitted matches every log.
#[serde(default)]
pub filter: Option<String>,
/// If true, return only how many logs matched instead of the log lines themselves.
#[serde(default)]
pub count_only: bool,
/// Maximum number of log lines to return. Ignored when `count_only` is set.
#[serde(default)]
pub limit: Option<usize>,
}
@@ -35,15 +22,11 @@ pub struct QueryLogsParams {
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
pub struct DeclareParamParams {
pub name: String,
/// Regex matched against each log line; `capture_group` extracts this param's value from it.
pub message_match_regex: String,
pub capture_group: String,
/// Dotted path into a JSON-structured log line to extract from instead of `capture_group`.
#[serde(default)]
pub json_field_path: Option<String>,
/// One of: "string", "number", "timestamp", "duration", "boolean".
pub kind: String,
/// Extract the elapsed time since this param was last seen, instead of parsing captured text.
#[serde(default)]
pub elapsed_since_last: bool,
}
@@ -52,23 +35,18 @@ pub struct DeclareParamParams {
pub struct DeclareMetricParams {
pub name: String,
pub message_match_regex: String,
/// Names of already-declared params whose values become this metric's bucket labels.
#[serde(default)]
pub label_params: Vec<String>,
/// Name of an already-declared param whose value becomes this metric's per-event bucket value.
#[serde(default)]
pub value_param: Option<String>,
/// Fixed labels layered on top of `label_params` for matching lines.
#[serde(default)]
pub matched_labels: Option<BTreeMap<String, String>>,
/// Fixed labels recorded once for a non-matching line, instead of it being invisible to this metric.
#[serde(default)]
pub unmatched_labels: Option<BTreeMap<String, String>>,
}
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct CreateDashboardParams {
/// Profile to create the dashboard in. Defaults to the currently active profile.
#[serde(default)]
pub profile: Option<String>,
pub name: String,
@@ -76,17 +54,13 @@ pub struct CreateDashboardParams {
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct AddPanelParams {
/// Profile the dashboard lives in. Defaults to the currently active profile.
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
/// Id of an existing panel to update in place; omit to add a new panel.
#[serde(default)]
pub panel_id: Option<String>,
pub title: String,
/// PromQL-style query backing this panel (see `crate::promql`).
pub promql: String,
/// One of: "bar", "line", "pie". Defaults to "line".
#[serde(default = "default_chart_type")]
pub chart_type: String,
}
@@ -97,7 +71,6 @@ fn default_chart_type() -> String {
#[derive(Debug, Deserialize, schemars::JsonSchema)]
pub struct GetDashboardParams {
/// Profile the dashboard lives in. Defaults to the currently active profile.
#[serde(default)]
pub profile: Option<String>,
pub dashboard_id: String,
@@ -107,9 +80,6 @@ pub struct GetDashboardParams {
pub struct OttylMcp {
client: reqwest::Client,
base_url: String,
// Constructed fresh per call by the `#[tool_handler]`-generated `ServerHandler` methods
// (`Self::tool_router()`), not read through this field — kept only because `#[tool_router]`
// requires a field of this name/type to exist on the struct.
#[allow(dead_code)]
tool_router: ToolRouter<OttylMcp>,
}
@@ -138,8 +108,6 @@ impl OttylMcp {
format!("{}{path}", self.base_url)
}
/// Resolves the profile a dashboard tool call should operate on: the explicit `profile`
/// argument if given, otherwise the target instance's currently active profile.
async fn resolve_profile(&self, explicit: Option<String>) -> Result<String, McpError> {
if let Some(profile) = explicit {
return Ok(profile);
@@ -282,10 +250,6 @@ impl OttylMcp {
json_result(body)
}
// Read-modify-write over HTTP (no dashboard-level locking): two concurrent `add_panel` calls
// against the same dashboard can race and one's addition can be lost, same as two browser tabs
// saving the same dashboard through the webapp today. Not addressed here — out of scope for a
// thin proxy over the existing `save_dashboard` endpoint.
#[tool(description = "Add a new panel to a dashboard, or update an existing one by panel_id.")]
async fn add_panel(
&self,
@@ -359,8 +323,6 @@ impl OttylMcp {
}
}
/// Percent-encodes a single path segment (profile/dashboard names/ids can contain characters that
/// aren't valid unescaped in a URL path).
fn urlencoding_path(segment: &str) -> String {
segment
.bytes()
@@ -373,8 +335,6 @@ fn urlencoding_path(segment: &str) -> String {
.collect()
}
/// Generates a new panel id by hashing title/promql/time, avoiding a `uuid` dependency (matches
/// `settings::generate_imported_css_id`'s approach for the same reason).
fn generate_panel_id(title: &str, promql: &str) -> String {
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
src/api.rs → src/api/mod.rs
+142 -217
View File
@@ -1,21 +1,24 @@
pub mod dto;
pub mod mcp_server;
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::dto::{
use crate::api::dto::{
BucketValuesParams, CreateAlertRuleRequest, CreateDashboardRequest, CreateProfileRequest,
CssEntryDto, DashboardDto, ErrorDto, ImportCssRequest, LinkCssRequest, MetricDeclarationDto,
ParamDeclarationDto, LogsParams, ProfileSummaryDto, PromqlParams, SaveDashboardRequest,
SettingsDto, TestAlertNotificationRequest, TimelineParams, UpdateAlertRuleRequest, UpdateSettingsRequest,
};
use crate::data_value::MetricPrimitiveKind;
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::{
DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage, EngineQuery,
EngineResponse,
};
use crate::alerts::{deliver_ntfy, AlertError};
use crate::profile::{self, Dashboard, ParamDeclarationRecord, Profile, ProfileError};
use crate::settings::{CssError, SettingsHandle};
use crate::engine::alerts::{deliver_ntfy, AlertError};
use crate::config::profile::{Dashboard, ParamDeclarationRecord, Profile, ProfileError};
use crate::config::{CssError, SettingsHandle};
use axum::extract::{Path as AxumPath, Query, State};
use axum::http::{header, StatusCode};
use axum::response::{IntoResponse, Json, Response};
@@ -26,15 +29,12 @@ use tokio::sync::oneshot;
static WEBAPP_DIST: EmbedDir = fs_embed::fs_embed!("webapp/dist");
/// Embedded so an agent that has only discovered a port (no filesystem access into the ottyl
/// repo/binary) can still fetch it — see `SKILL.md` at the repo root and ticket 069.
const SKILL_MD: &str = include_str!("../SKILL.md");
const SKILL_MD: &str = include_str!("../../SKILL.md");
#[derive(Clone)]
struct ApiState {
engine: EngineHandle,
settings: SettingsHandle,
/// Imported CSS files are stored alongside this path's parent directory (see `ApiState::css_dir`).
settings_path: Arc<PathBuf>,
}
@@ -46,7 +46,6 @@ impl ApiState {
.unwrap_or_else(|| PathBuf::from("."))
}
/// Best-effort: a no-op if no profile is active, and I/O failures are only logged, since the engine declaration is already the source of truth.
fn persist_param_declaration(&self, request: &DeclareParamRequest) {
let mut settings = self.settings.lock().unwrap();
let Some(profile_name) = settings.active_profile.clone() else {
@@ -60,11 +59,6 @@ impl ApiState {
}
}
/// Counterpart to `persist_param_declaration` for `DELETE /api/params` (ticket 052): empties the
/// active profile's stored params so the just-cleared set doesn't reappear from `settings.json`
/// on next load. Same best-effort semantics — a no-op if no profile is active, I/O failures only
/// logged, since the engine-side clear (already applied by the time this runs) is the source of
/// truth for the running process.
fn persist_clear_params(&self) {
let mut settings = self.settings.lock().unwrap();
let Some(profile_name) = settings.active_profile.clone() else {
@@ -77,11 +71,6 @@ impl ApiState {
}
}
/// Same as `persist_param_declaration`, for metric declarations. Also promotes any of the
/// metric's referenced params that only exist as auto-derived engine state (see
/// `promoted_param_records`) into persisted profile params, so a metric built against an
/// auto-detected logfmt key survives a restart instead of being rejected as "undeclared" once
/// declarations replay before any log line re-derives it.
async fn persist_metric_declaration(&self, request: &DeclareMetricRequest) {
let profile_name = {
let settings = self.settings.lock().unwrap();
@@ -109,7 +98,6 @@ impl ApiState {
}
}
/// Same as `persist_metric_declaration`, for an in-place edit (`PUT /api/metrics/{name}`) — also drops the persisted record under `old_name` when the edit renames the metric, so a rename doesn't leave a stale duplicate declaration in the profile file.
async fn persist_metric_declaration_update(&self, old_name: &str, request: &DeclareMetricRequest) {
let profile_name = {
let settings = self.settings.lock().unwrap();
@@ -138,9 +126,6 @@ impl ApiState {
}
}
/// Best-effort, matching `persist_param_declaration`'s intent — a no-op if no profile is
/// active, and I/O failures are only logged, since the engine's removal is already the source
/// of truth for what the live process shows.
fn persist_metric_declaration_removal(&self, name: &str) {
let mut settings = self.settings.lock().unwrap();
let Some(profile_name) = settings.active_profile.clone() else {
@@ -203,7 +188,6 @@ async fn get_skill_md() -> impl IntoResponse {
([(header::CONTENT_TYPE, "text/markdown; charset=utf-8")], SKILL_MD)
}
/// Falls back to `index.html` for unmatched paths so client-side routing works.
async fn serve_webapp(uri: axum::http::Uri) -> Response {
let path = uri.path().trim_start_matches('/');
let file = WEBAPP_DIST
@@ -234,22 +218,28 @@ fn mime_for_extension(extension: &str) -> &'static str {
}
}
async fn ask(engine: &EngineHandle, query: EngineQuery) -> Result<EngineResponse, StatusCode> {
let (respond_to, response) = oneshot::channel();
engine
.send(EngineMessage::Query { query, respond_to })
.await
.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;
response.await.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)
impl EngineHandle {
async fn ask(&self, query: EngineQuery) -> Result<EngineResponse, StatusCode> {
let read_handle = self.reader.handle();
if let Some(sort_by) = query.sort_by() {
let needs_build = match read_handle.enter() {
Some(guard) => !guard.has_sort_helper_index(sort_by),
None => return Err(StatusCode::SERVICE_UNAVAILABLE),
};
if needs_build {
let (respond_to, response) = oneshot::channel();
self.tx
.send(EngineMessage::EnsureSortHelperIndex { param_name: sort_by.to_string(), respond_to })
.await
.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;
response.await.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;
}
}
let guard = read_handle.enter().ok_or(StatusCode::SERVICE_UNAVAILABLE)?;
Ok(guard.handle_query_ro(query))
}
}
/// Any of `request`'s referenced params (`label_params`/`value_param`) not already present in
/// `existing_params` are looked up live in the engine and returned as records ready to persist
/// alongside the metric (see `ProfileStore::record_metric_declaration`'s `promoted_params`). A
/// metric can validly reference a param that's only auto-derived engine state (see
/// `crate::engine::auto_declare_params`) — that param was never written into any profile, so without
/// this promotion the metric would be rejected as "undeclared" the next time declarations replay on
/// startup, before any log line has re-derived it.
async fn promoted_param_records(
engine: &EngineHandle,
existing_params: &[ParamDeclarationRecord],
@@ -266,7 +256,7 @@ async fn promoted_param_records(
return Vec::new();
}
let Ok(EngineResponse::ParamDeclarations(declarations)) =
ask(engine, EngineQuery::ParamDeclarations).await
engine.ask(EngineQuery::ParamDeclarations).await
else {
return Vec::new();
};
@@ -288,7 +278,7 @@ async fn get_logs(
State(state): State<ApiState>,
Query(params): Query<LogsParams>,
) -> impl IntoResponse {
match ask(&state.engine, EngineQuery::Logs(params)).await {
match state.engine.ask(EngineQuery::Logs(params)).await {
Ok(EngineResponse::Logs(entries)) => Json(entries).into_response(),
Ok(EngineResponse::Error(err)) => {
(StatusCode::BAD_REQUEST, Json(ErrorDto { error: err })).into_response()
@@ -302,7 +292,7 @@ async fn get_timeline(
State(state): State<ApiState>,
Query(params): Query<TimelineParams>,
) -> impl IntoResponse {
match ask(&state.engine, EngineQuery::Timeline(params)).await {
match state.engine.ask(EngineQuery::Timeline(params)).await {
Ok(EngineResponse::Timeline(buckets)) => Json(buckets).into_response(),
Ok(EngineResponse::Error(err)) => {
(StatusCode::BAD_REQUEST, Json(ErrorDto { error: err })).into_response()
@@ -313,7 +303,7 @@ async fn get_timeline(
}
async fn get_stats(State(state): State<ApiState>) -> impl IntoResponse {
match ask(&state.engine, EngineQuery::Stats).await {
match state.engine.ask(EngineQuery::Stats).await {
Ok(EngineResponse::Stats(stats)) => Json(stats).into_response(),
Ok(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
Err(status) => status.into_response(),
@@ -321,16 +311,9 @@ async fn get_stats(State(state): State<ApiState>) -> impl IntoResponse {
}
async fn list_params(State(state): State<ApiState>) -> impl IntoResponse {
match ask(&state.engine, EngineQuery::ParamDeclarations).await {
match state.engine.ask(EngineQuery::ParamDeclarations).await {
Ok(EngineResponse::ParamDeclarations(mut declarations)) => {
// `ottyl_seq`/`ottyl_ts`/`ottyl_fmt` (ticket 055) are computed line-locally rather than
// stored as index `ParamDeclaration` entries (see `crate::engine::declare_param`'s
// rejection of these names), so they're merged in here purely for discoverability
// rather than via `param_declarations_range`. This is deliberately a merge only in this
// one HTTP-facing listing, not in `EngineQuery::ParamDeclarations` itself — other
// internal callers of that query (promoting a metric's referenced params, replaying
// declarations on profile switch) must keep seeing only real, persistable entries.
declarations.extend(reserved_param_declaration_dtos());
declarations.extend(ParamDeclarationDto::reserved());
Json(declarations).into_response()
}
Ok(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
@@ -338,43 +321,41 @@ async fn list_params(State(state): State<ApiState>) -> impl IntoResponse {
}
}
/// `GET /api/params`-only synthesized entries for the three reserved names
/// (`crate::engine::OTTYL_RESERVED_PARAM_NAMES`) — see `list_params`. `message_match_regex`/
/// `capture_group` are empty since these are never text-extracted; `reserved: true` marks them as
/// non-deletable/non-editable, distinct from `auto_derived`.
fn reserved_param_declaration_dtos() -> [ParamDeclarationDto; 3] {
[
ParamDeclarationDto {
name: "ottyl_seq".to_string(),
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: None,
kind: MetricPrimitiveKind::Number,
elapsed_since_last: false,
auto_derived: false,
reserved: true,
},
ParamDeclarationDto {
name: "ottyl_ts".to_string(),
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: None,
kind: MetricPrimitiveKind::Timestamp,
elapsed_since_last: false,
auto_derived: false,
reserved: true,
},
ParamDeclarationDto {
name: "ottyl_fmt".to_string(),
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: None,
kind: MetricPrimitiveKind::String,
elapsed_since_last: false,
auto_derived: false,
reserved: true,
},
]
impl ParamDeclarationDto {
fn reserved() -> [ParamDeclarationDto; 3] {
[
ParamDeclarationDto {
name: "ottyl_seq".to_string(),
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: None,
kind: MetricPrimitiveKind::Number,
elapsed_since_last: false,
auto_derived: false,
reserved: true,
},
ParamDeclarationDto {
name: "ottyl_ts".to_string(),
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: None,
kind: MetricPrimitiveKind::Timestamp,
elapsed_since_last: false,
auto_derived: false,
reserved: true,
},
ParamDeclarationDto {
name: "ottyl_fmt".to_string(),
message_match_regex: String::new(),
capture_group: String::new(),
json_field_path: None,
kind: MetricPrimitiveKind::String,
elapsed_since_last: false,
auto_derived: false,
reserved: true,
},
]
}
}
async fn declare_param(
@@ -385,6 +366,7 @@ async fn declare_param(
let request = body.clone();
if state
.engine
.tx
.send(EngineMessage::DeclareParam { request: body, respond_to })
.await
.is_err()
@@ -401,8 +383,6 @@ async fn declare_param(
json_field_path: request.json_field_path,
kind: request.kind,
elapsed_since_last: request.elapsed_since_last,
// A request that reached here was accepted by `declare_param`, which always stores
// it as non-auto (see `engine::declare_param`).
auto_derived: false,
reserved: false,
};
@@ -413,14 +393,9 @@ async fn declare_param(
}
}
/// `DELETE /api/params` (ticket 052): clears every declared param (manual and auto-derived) and
/// immediately re-derives whatever `auto_declare_params` can still find in log text the engine
/// still retains. The response body's `logs_considered` tells the caller how many retained lines
/// were re-scanned, so the UI can make clear that lines evicted by `max_logs` before this ran
/// can't regain their params — this is expected behavior, not a partial failure.
async fn clear_params(State(state): State<ApiState>) -> impl IntoResponse {
let (respond_to, response) = oneshot::channel();
if state.engine.send(EngineMessage::ClearParams { respond_to }).await.is_err() {
if state.engine.tx.send(EngineMessage::ClearParams { respond_to }).await.is_err() {
return StatusCode::SERVICE_UNAVAILABLE.into_response();
}
match response.await {
@@ -433,7 +408,7 @@ async fn clear_params(State(state): State<ApiState>) -> impl IntoResponse {
}
async fn list_metrics(State(state): State<ApiState>) -> impl IntoResponse {
match ask(&state.engine, EngineQuery::MetricDeclarations).await {
match state.engine.ask(EngineQuery::MetricDeclarations).await {
Ok(EngineResponse::MetricDeclarations(declarations)) => Json(declarations).into_response(),
Ok(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
Err(status) => status.into_response(),
@@ -448,6 +423,7 @@ async fn declare_metric(
let request = body.clone();
if state
.engine
.tx
.send(EngineMessage::DeclareMetric { request: body, respond_to })
.await
.is_err()
@@ -472,7 +448,6 @@ async fn declare_metric(
}
}
/// Updates the metric declaration named by the URL path in place, reusing `declare_metric`'s validation (via `engine::update_metric`) rather than creating a second declaration or erroring on a duplicate-name conflict. `body.name` may differ from the path segment to rename the metric, so long as no other declaration already uses the new name.
async fn update_metric(
State(state): State<ApiState>,
AxumPath(name): AxumPath<String>,
@@ -482,6 +457,7 @@ async fn update_metric(
let request = body.clone();
if state
.engine
.tx
.send(EngineMessage::UpdateMetric { name: name.clone(), request: body, respond_to })
.await
.is_err()
@@ -506,12 +482,6 @@ async fn update_metric(
}
}
/// Deletes the metric declaration named by the URL path — both from the live engine index and,
/// if a profile is active, from persisted settings (ticket 051), so it doesn't reappear on
/// restart. Its recorded bucket-value history is purged along with it (see the comment on
/// `EngineMessage::RemoveMetricDeclaration`'s handler in `engine.rs`), rather than left orphaned,
/// so a dashboard panel or PromQL query still referencing the metric by name fails with a clear
/// "unknown metric" error instead of silently keeping stale data alive.
async fn delete_metric(
State(state): State<ApiState>,
AxumPath(name): AxumPath<String>,
@@ -519,6 +489,7 @@ async fn delete_metric(
let (respond_to, response) = oneshot::channel();
if state
.engine
.tx
.send(EngineMessage::RemoveMetricDeclaration { name: name.clone(), respond_to })
.await
.is_err()
@@ -535,7 +506,6 @@ async fn delete_metric(
}
}
/// Malformed pairs (no `=`) are skipped rather than rejected.
fn parse_labels_filter(raw: &str) -> BTreeMap<String, String> {
raw.split(',')
.filter_map(|pair| pair.split_once('='))
@@ -557,7 +527,7 @@ async fn get_metric_bucket_values(
since_sequence: params.since_sequence,
limit: params.limit,
};
match ask(&state.engine, query).await {
match state.engine.ask(query).await {
Ok(EngineResponse::BucketValues(values)) => Json(values).into_response(),
Ok(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(),
Err(status) => status.into_response(),
@@ -573,7 +543,7 @@ async fn run_promql_query(
start_ms: params.start_ms,
end_ms: params.end_ms,
};
match ask(&state.engine, query).await {
match state.engine.ask(query).await {
Ok(EngineResponse::Promql(Ok(series))) => Json(series).into_response(),
Ok(EngineResponse::Promql(Err(err))) => {
(StatusCode::BAD_REQUEST, Json(ErrorDto { error: err })).into_response()
@@ -583,11 +553,13 @@ async fn run_promql_query(
}
}
fn css_error_response(err: CssError) -> axum::response::Response {
match err {
CssError::NotFound => (StatusCode::NOT_FOUND, err.to_string()).into_response(),
CssError::NotImported => (StatusCode::BAD_REQUEST, err.to_string()).into_response(),
CssError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(),
impl CssError {
fn into_response(self) -> axum::response::Response {
match self {
CssError::NotFound => (StatusCode::NOT_FOUND, self.to_string()).into_response(),
CssError::NotImported => (StatusCode::BAD_REQUEST, self.to_string()).into_response(),
CssError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response(),
}
}
}
@@ -608,15 +580,14 @@ async fn import_css(
let entry = settings
.css_entries
.iter()
.find(|entry| crate::settings::css_entry_id(entry) == id)
.find(|entry| crate::config::css_entry_id(entry) == id)
.expect("entry was just inserted by import_css");
(StatusCode::CREATED, Json(CssEntryDto::from(entry))).into_response()
}
Err(err) => css_error_response(err),
Err(err) => err.into_response(),
}
}
/// Content is never copied, and the entry can never be deleted via `DELETE /api/settings/css/:id`.
async fn link_css(
State(state): State<ApiState>,
Json(body): Json<LinkCssRequest>,
@@ -630,7 +601,7 @@ async fn link_css(
.expect("entry was just pushed by link_css");
(StatusCode::CREATED, Json(CssEntryDto::from(entry))).into_response()
}
Err(err) => css_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -642,11 +613,10 @@ async fn remove_css(
let mut settings = state.settings.lock().unwrap();
match settings.remove_css(&state.settings_path, &css_dir, &id) {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(err) => css_error_response(err),
Err(err) => err.into_response(),
}
}
/// Rejects a linked entry with 400 — there's no local content; the frontend should use `path_or_url` directly.
async fn get_css_content(
State(state): State<ApiState>,
AxumPath(id): AxumPath<String>,
@@ -660,14 +630,16 @@ async fn get_css_content(
content,
)
.into_response(),
Err(err) => css_error_response(err),
Err(err) => err.into_response(),
}
}
fn alert_error_response(err: AlertError) -> axum::response::Response {
match err {
AlertError::NotFound => (StatusCode::NOT_FOUND, err.to_string()).into_response(),
AlertError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(),
impl AlertError {
fn into_response(self) -> axum::response::Response {
match self {
AlertError::NotFound => (StatusCode::NOT_FOUND, self.to_string()).into_response(),
AlertError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response(),
}
}
}
@@ -683,7 +655,7 @@ async fn create_alert_rule(
let mut settings = state.settings.lock().unwrap();
match settings.create_alert_rule(&state.settings_path, body.name, body.enabled, body.condition) {
Ok(rule) => (StatusCode::CREATED, Json(rule)).into_response(),
Err(err) => alert_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -695,7 +667,7 @@ async fn update_alert_rule(
let mut settings = state.settings.lock().unwrap();
match settings.update_alert_rule(&state.settings_path, &id, body.name, body.enabled, body.condition) {
Ok(rule) => Json(rule).into_response(),
Err(err) => alert_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -703,11 +675,10 @@ async fn delete_alert_rule(State(state): State<ApiState>, AxumPath(id): AxumPath
let mut settings = state.settings.lock().unwrap();
match settings.remove_alert_rule(&state.settings_path, &id) {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(err) => alert_error_response(err),
Err(err) => err.into_response(),
}
}
/// Delivers a test notification without a saved rule so the sink can be verified before saving.
async fn test_alert_notification(Json(body): Json<TestAlertNotificationRequest>) -> impl IntoResponse {
let server = body.server.unwrap_or_else(|| "https://ntfy.sh".to_string());
match deliver_ntfy(&server, &body.topic, "Test notification from ottyl").await {
@@ -716,13 +687,11 @@ async fn test_alert_notification(Json(body): Json<TestAlertNotificationRequest>)
}
}
/// CSS entries have their own endpoint (`GET /api/settings/css`).
async fn get_settings(State(state): State<ApiState>) -> impl IntoResponse {
let settings = state.settings.lock().unwrap();
Json(SettingsDto::from(&*settings)).into_response()
}
/// `max_logs: null` clears the cap; a changed value is pushed to the engine so it applies without a restart.
async fn update_settings(
State(state): State<ApiState>,
Json(body): Json<UpdateSettingsRequest>,
@@ -754,18 +723,22 @@ async fn update_settings(
if max_logs_changed {
let _ = state
.engine
.tx
.send(EngineMessage::SetMaxLogs(new_max_logs.1))
.await;
state.engine.barrier().await;
}
Json(new_max_logs.0).into_response()
}
fn profile_error_response(err: ProfileError) -> axum::response::Response {
match err {
ProfileError::NotFound => (StatusCode::NOT_FOUND, err.to_string()).into_response(),
ProfileError::DuplicateName => (StatusCode::BAD_REQUEST, err.to_string()).into_response(),
ProfileError::CannotDeleteDefault => (StatusCode::BAD_REQUEST, err.to_string()).into_response(),
ProfileError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(),
impl ProfileError {
fn into_response(self) -> axum::response::Response {
match self {
ProfileError::NotFound => (StatusCode::NOT_FOUND, self.to_string()).into_response(),
ProfileError::DuplicateName => (StatusCode::BAD_REQUEST, self.to_string()).into_response(),
ProfileError::CannotDeleteDefault => (StatusCode::BAD_REQUEST, self.to_string()).into_response(),
ProfileError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response(),
}
}
}
@@ -785,11 +758,10 @@ async fn create_profile(
let profile = settings.profile(&body.name).expect("profile was just created by create_profile");
(StatusCode::CREATED, Json(ProfileSummaryDto::from(profile))).into_response()
}
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
/// If it was the active profile, `active_profile` is cleared, but the engine's already-declared params/metrics keep running until the next restart.
async fn delete_profile(
State(state): State<ApiState>,
AxumPath(name): AxumPath<String>,
@@ -797,21 +769,18 @@ async fn delete_profile(
let mut settings = state.settings.lock().unwrap();
match settings.delete_profile(&state.settings_path, &name) {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
/// Removes every currently-declared param/metric from the engine, then replays `profile`'s declarations into it.
/// Auto-derived params (`ParamDeclarationDto::auto_derived`) are left alone — they aren't part of
/// any profile, just facts discovered from the log stream, so a profile switch shouldn't discard
/// them (they'd otherwise only reappear once a new line re-introduces that key).
async fn reload_engine_declarations(engine: &EngineHandle, profile: &Profile) {
if let Ok(EngineResponse::ParamDeclarations(params)) =
ask(engine, EngineQuery::ParamDeclarations).await
engine.ask(EngineQuery::ParamDeclarations).await
{
for param in params.into_iter().filter(|param| !param.auto_derived) {
let (respond_to, response) = oneshot::channel();
if engine
.tx
.send(EngineMessage::RemoveParamDeclaration { name: param.name, respond_to })
.await
.is_ok()
@@ -821,11 +790,12 @@ async fn reload_engine_declarations(engine: &EngineHandle, profile: &Profile) {
}
}
if let Ok(EngineResponse::MetricDeclarations(metrics)) =
ask(engine, EngineQuery::MetricDeclarations).await
engine.ask(EngineQuery::MetricDeclarations).await
{
for metric in metrics {
let (respond_to, response) = oneshot::channel();
if engine
.tx
.send(EngineMessage::RemoveMetricDeclaration { name: metric.name, respond_to })
.await
.is_ok()
@@ -834,7 +804,7 @@ async fn reload_engine_declarations(engine: &EngineHandle, profile: &Profile) {
}
}
}
profile::load_into_engine(engine, profile).await;
profile.load_into_engine(engine).await;
}
async fn activate_profile(
@@ -857,7 +827,7 @@ async fn activate_profile(
let settings = state.settings.lock().unwrap();
Json(SettingsDto::from(&*settings)).into_response()
}
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -871,7 +841,7 @@ async fn list_dashboards(
let dtos: Vec<DashboardDto> = dashboards.iter().map(DashboardDto::from).collect();
Json(dtos).into_response()
}
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -883,7 +853,7 @@ async fn create_dashboard(
let mut settings = state.settings.lock().unwrap();
match settings.create_dashboard(&state.settings_path, &name, body.name) {
Ok(dashboard) => (StatusCode::CREATED, Json(DashboardDto::from(&dashboard))).into_response(),
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -895,9 +865,9 @@ async fn get_dashboard(
match settings.list_dashboards(&name) {
Ok(dashboards) => match dashboards.iter().find(|d| d.id == dashboard_id) {
Some(dashboard) => Json(DashboardDto::from(dashboard)).into_response(),
None => profile_error_response(ProfileError::NotFound),
None => ProfileError::NotFound.into_response(),
},
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -914,7 +884,7 @@ async fn save_dashboard(
};
match settings.save_dashboard(&state.settings_path, &name, &dashboard_id, dashboard) {
Ok(dashboard) => Json(DashboardDto::from(&dashboard)).into_response(),
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -925,7 +895,7 @@ async fn delete_dashboard(
let mut settings = state.settings.lock().unwrap();
match settings.delete_dashboard(&state.settings_path, &name, &dashboard_id) {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
Err(err) => profile_error_response(err),
Err(err) => err.into_response(),
}
}
@@ -933,25 +903,21 @@ async fn delete_dashboard(
mod tests {
use super::*;
use crate::engine::{self, IngestedLine};
use crate::settings::Settings;
use crate::timestamp::Timestamp;
use crate::config::Settings;
use crate::engine::timestamp::Timestamp;
use axum::body::Body;
use axum::http::Request;
use http_body_util::BodyExt;
use serde_json::{json, Value};
use std::sync::Mutex;
use tokio::sync::mpsc;
use tower::ServiceExt;
/// Spins up a fresh engine + router pair backed by a temp settings file.
fn test_app() -> (Router, EngineHandle, SettingsHandle, tempfile::TempDir) {
let (engine_tx, engine_rx) = mpsc::channel(128);
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
tokio::spawn(engine::run(engine_rx, settings.clone()));
let engine_tx = engine::spawn(settings.clone(), 128);
let app = router(engine_tx.clone(), settings.clone(), settings_path);
(app, engine_tx, settings, dir)
@@ -1039,7 +1005,6 @@ mod tests {
let status = delete_request(&app, "/api/settings/alerts/does-not-exist").await;
assert_eq!(status, StatusCode::NOT_FOUND);
// AlertError::NotFound maps to a plain-text (not JSON) 404 body, so check status directly instead of via send_json.
let request = Request::builder()
.method("PUT")
.uri("/api/settings/alerts/does-not-exist")
@@ -1077,10 +1042,6 @@ mod tests {
assert_eq!(body["capture_group"], "level");
}
/// Ticket 052's core acceptance criteria: `DELETE /api/params` removes every declared param
/// (manual and auto-derived), that removal survives a reload of persisted settings, and —
/// without any new line being ingested — the retained `IngestedLog` entries immediately
/// re-derive a fresh (auto-derived) param set.
#[tokio::test]
async fn clearing_params_persists_and_reextracts_from_retained_logs() {
let (app, engine, settings_handle, dir) = test_app();
@@ -1092,14 +1053,10 @@ mod tests {
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
}
// A manual declaration plus an auto-derived one (from ingesting a logfmt line before "path" is declared).
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
ingest_lines(&engine, 0, &["request path=/a", "level=info x=1"]).await;
let (_, before) = get_json(&app, "/api/params").await;
// Ticket 055 merges the three non-deletable `ottyl_*` reserved names into this listing
// too; `DELETE /api/params` (ticket 052) only ever clears real declarations, so the
// reserved ones must be excluded from the "everything got removed" count below.
let before_declared: Vec<&str> = before
.as_array()
.unwrap()
@@ -1114,7 +1071,6 @@ mod tests {
let (status, result) = send_json(&app, "DELETE", "/api/params", json!({})).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(result["removed"], before_declared.len() as u64);
// Re-derived without any new ingest — same three keys come back, now all auto-derived.
assert_eq!(result["reextracted"], 3);
assert_eq!(result["logs_considered"], 2);
@@ -1124,8 +1080,6 @@ mod tests {
assert_eq!(after_declared.len(), 3);
assert!(after_declared.iter().all(|p| p["auto_derived"] == true));
// The clear persisted: "path" (the only manually-declared, profile-stored param) doesn't
// reappear from settings.json on a fresh load.
let reloaded_settings = Settings::load(&settings_path);
let profile = reloaded_settings.profile("web").unwrap();
assert!(profile.params.is_empty());
@@ -1150,9 +1104,6 @@ mod tests {
assert!(body["error"].as_str().unwrap().contains("invalid regex"));
}
/// Declares a standalone param via `POST /api/params` — most metric-declaration tests need one
/// or more already-declared params to reference by name (ticket 037), rather than embedding a
/// regex/capture-group/kind inline in the metric request.
async fn declare_test_param(app: &Router, name: &str, pattern: &str, capture_group: &str, kind: &str) {
let (status, _body) = send_json(
app,
@@ -1244,7 +1195,6 @@ mod tests {
assert_eq!(body["name"], "requests");
assert_eq!(body["label_params"].as_array().unwrap().len(), 2);
// Still exactly one declaration named "requests", now with the updated regex/params — not a second declaration alongside the original.
let (status, body) = get_json(&app, "/api/metrics").await;
assert_eq!(status, StatusCode::OK);
let metrics = body.as_array().unwrap();
@@ -1330,8 +1280,6 @@ mod tests {
assert_eq!(status, StatusCode::NOT_FOUND);
}
/// Acceptance criterion: deleting a metric persists the removal into the active profile too, so
/// it doesn't reappear from `settings.json` on the next restart.
#[tokio::test]
async fn deleting_metric_persists_removal_and_it_does_not_reappear_after_restart() {
let (app, _engine, settings_handle, dir) = test_app();
@@ -1362,10 +1310,6 @@ mod tests {
assert!(Settings::load(&settings_path).profile("web").unwrap().metrics.is_empty());
}
/// Acceptance criterion: a dashboard panel (PromQL query) that still references a metric by
/// name after it's been deleted fails with a readable "unknown metric" error, rather than a
/// panic or a silently-empty series — seeded here by declaring, ingesting into, deleting, then
/// re-querying the same metric name.
#[tokio::test]
async fn promql_query_against_deleted_metric_fails_with_readable_error() {
let (app, engine, _settings, _dir) = test_app();
@@ -1383,6 +1327,7 @@ mod tests {
)
.await;
engine
.tx
.send(EngineMessage::Ingest(IngestedLine {
sequence: 0,
timestamp: Timestamp::from_millis(0),
@@ -1390,8 +1335,8 @@ mod tests {
}))
.await
.unwrap();
engine.barrier().await;
// Sanity check: the dashboard-panel-style query works before deletion.
let (status, _body) = get_json(&app, "/api/query?promql=requests").await;
assert_eq!(status, StatusCode::OK);
@@ -1429,6 +1374,7 @@ mod tests {
(2u64, "request path=/a"),
] {
engine
.tx
.send(EngineMessage::Ingest(IngestedLine {
sequence,
timestamp: Timestamp::from_millis(sequence as i64 * 1000),
@@ -1437,6 +1383,7 @@ mod tests {
.await
.unwrap();
}
engine.barrier().await;
let (status, body) = get_json(&app, "/api/metrics/requests").await;
assert_eq!(status, StatusCode::OK);
@@ -1447,11 +1394,9 @@ mod tests {
let values = body.as_array().unwrap();
assert_eq!(values.len(), 2);
for value in values {
// MetricPrimitive derives Serialize as an externally-tagged enum, so JSON is {"String": "/a"}.
assert_eq!(value["labels"]["path"]["String"], "/a");
}
// Bucket values group by label (BTreeMap-ordered "/a" before "/b"), not by sequence, so limit=1 picks sequence 2 from "/a" ahead of sequence 1 from "/b".
let (status, body) = get_json(&app, "/api/metrics/requests?since_sequence=0&limit=1").await;
assert_eq!(status, StatusCode::OK);
let values = body.as_array().unwrap();
@@ -1482,6 +1427,7 @@ mod tests {
(2u64, "request path=/a"),
] {
engine
.tx
.send(EngineMessage::Ingest(IngestedLine {
sequence,
timestamp: Timestamp::from_millis(sequence as i64 * 1000),
@@ -1490,6 +1436,7 @@ mod tests {
.await
.unwrap();
}
engine.barrier().await;
let (status, body) =
get_json(&app, "/api/query?promql=requests%7Bpath%3D%22%2Fa%22%7D").await;
@@ -1515,7 +1462,6 @@ mod tests {
async fn declaring_param_persists_into_active_profile() {
let (app, _engine, settings_handle, dir) = test_app();
// Use the same shared settings handle the router was built with — a separately-loaded Settings wouldn't be visible to the running app.
let settings_path = dir.path().join("settings.json");
{
let mut settings = settings_handle.lock().unwrap();
@@ -1543,11 +1489,6 @@ mod tests {
assert_eq!(profile.params[0].name, "level");
}
/// Regression test: declaring a metric against a param that only exists as auto-derived engine
/// state (discovered from an ingested logfmt key, never manually declared via `POST
/// /api/params`) must persist that param into the active profile too, not just the metric —
/// otherwise a restart replays the metric before any log line re-derives the auto param, and the
/// engine rejects it as referencing an undeclared param (see `promoted_param_records`).
#[tokio::test]
async fn declaring_metric_against_auto_derived_param_promotes_it_into_the_profile() {
let (app, engine, settings_handle, dir) = test_app();
@@ -1559,8 +1500,6 @@ mod tests {
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
}
// Ingest a line carrying a logfmt key before any param is manually declared, so "dx" only
// exists as engine-internal auto-derived state (never persisted anywhere).
ingest_lines(&engine, 0, &["dx=1 moved"]).await;
let (status, _body) = send_json(
@@ -1580,22 +1519,15 @@ mod tests {
let profile = reloaded_settings.profile("web").unwrap();
assert_eq!(profile.metrics.len(), 1);
assert_eq!(profile.metrics[0].name, "direction_x");
// The auto-derived "dx" param got promoted into the profile alongside the metric.
assert_eq!(profile.params.len(), 1);
assert_eq!(profile.params[0].name, "dx");
// Simulate a restart: replay the persisted profile into a fresh engine with no prior ingestion.
let (fresh_engine_tx, fresh_engine_rx) = mpsc::channel(16);
let fresh_settings_handle = std::sync::Arc::new(std::sync::Mutex::new(Settings::default()));
tokio::spawn(crate::engine::run(fresh_engine_rx, fresh_settings_handle));
profile::load_into_engine(&fresh_engine_tx, profile).await;
let fresh_engine_tx = crate::engine::spawn(fresh_settings_handle, 16);
profile.load_into_engine(&fresh_engine_tx).await;
let (respond_to, response) = oneshot::channel();
fresh_engine_tx
.send(EngineMessage::Query { query: EngineQuery::MetricDeclarations, respond_to })
.await
.unwrap();
let EngineResponse::MetricDeclarations(metrics) = response.await.unwrap() else {
let response = fresh_engine_tx.ask(EngineQuery::MetricDeclarations).await.unwrap();
let EngineResponse::MetricDeclarations(metrics) = response else {
panic!("unexpected response");
};
assert_eq!(metrics.len(), 1);
@@ -1706,7 +1638,6 @@ mod tests {
)
.await;
// Simulate a restart by reloading settings fresh from disk, independent of the running app/engine.
let settings_path = dir.path().join("settings.json");
let reloaded_settings = Settings::load(&settings_path);
let dashboards = reloaded_settings.list_dashboards("web").unwrap();
@@ -1720,6 +1651,7 @@ mod tests {
for (offset, text) in texts.iter().enumerate() {
let sequence = start_sequence + offset as u64;
engine
.tx
.send(EngineMessage::Ingest(IngestedLine {
sequence,
timestamp: Timestamp::from_millis(sequence as i64 * 1000),
@@ -1728,6 +1660,7 @@ mod tests {
.await
.unwrap();
}
engine.barrier().await;
}
#[tokio::test]
@@ -1743,8 +1676,6 @@ mod tests {
assert_eq!(status, StatusCode::OK);
assert_eq!(body["max_logs"], 2);
// Existing over-cap logs are evicted as soon as the setting change
// reaches the engine, not just on the next ingest.
let (_, body) = get_json(&app, "/api/logs").await;
assert_eq!(body.as_array().unwrap().len(), 2);
@@ -1789,8 +1720,6 @@ mod tests {
assert_eq!(body.as_array().unwrap().len(), 3);
}
/// Ticket 045: seeded fixture mixing lines that carry the sort param with lines that don't,
/// exercised through the actual HTTP endpoint rather than just `handle_query` directly.
#[tokio::test]
async fn get_logs_sort_by_orders_by_param_and_toggles_missing_param_handling() {
let (app, engine, _settings, _dir) = test_app();
@@ -1801,28 +1730,24 @@ mod tests {
&engine,
0,
&[
"priority=3 boom", // sequence 0, has param
"no param here", // sequence 1, missing
"priority=1 boom", // sequence 2, has param
"also missing", // sequence 3, missing
"priority=3 boom",
"no param here",
"priority=1 boom",
"also missing",
],
)
.await;
// Default order (no sort_by): unchanged, plain ingest/ottyl_seq order.
let (status, body) = get_json(&app, "/api/logs").await;
assert_eq!(status, StatusCode::OK);
let sequences: Vec<u64> = body.as_array().unwrap().iter().map(|e| e["sequence"].as_u64().unwrap()).collect();
assert_eq!(sequences, vec![0, 1, 2, 3]);
// sort_by=priority with the fallback toggle on (the default): entries carrying the param
// sorted ascending by its value, then the missing-param entries ordered by ottyl_seq.
let (status, body) = get_json(&app, "/api/logs?sort_by=priority").await;
assert_eq!(status, StatusCode::OK);
let sequences: Vec<u64> = body.as_array().unwrap().iter().map(|e| e["sequence"].as_u64().unwrap()).collect();
assert_eq!(sequences, vec![2, 0, 1, 3]);
// Same sort_by, but with the fallback toggle explicitly off: missing-param lines are dropped.
let (status, body) = get_json(&app, "/api/logs?sort_by=priority&sort_missing_fallback=false").await;
assert_eq!(status, StatusCode::OK);
let sequences: Vec<u64> = body.as_array().unwrap().iter().map(|e| e["sequence"].as_u64().unwrap()).collect();
src/cli/mod.rs
+205
View File
@@ -0,0 +1,205 @@
pub mod sessions;
use std::path::PathBuf;
use std::sync::{Arc, LazyLock, Mutex};
use clap::{Parser, Subcommand};
use tokio::net::TcpListener;
use crate::config::Settings;
static SETTINGS_PATH_HELP: LazyLock<String> = LazyLock::new(|| {
format!(
"Path to the settings file. Defaults to {}.",
crate::config::default_settings_path()
.map(|path| path.display().to_string())
.unwrap_or_else(|| "a per-OS config location".to_string())
)
});
#[derive(Parser, Debug)]
struct Args {
#[command(subcommand)]
command: Option<Command>,
#[arg(long, default_value_t = 0)]
port: u16,
#[arg(long, help = SETTINGS_PATH_HELP.as_str())]
settings_path: Option<PathBuf>,
#[arg(long)]
file: Option<PathBuf>,
#[arg(long)]
metrics_url: Option<String>,
#[arg(long)]
name: Option<String>,
}
#[derive(Subcommand, Debug)]
enum Command {
Mcp {
#[arg(long)]
session: Option<String>,
#[arg(long)]
port: Option<u16>,
},
Sessions,
}
pub async fn run() {
let args = Args::parse();
let settings_path = args.settings_path.clone().unwrap_or_else(|| {
crate::config::default_settings_path()
.expect("could not determine a default settings path; pass --settings-path")
});
match args.command {
Some(Command::Mcp { session, port }) => {
run_mcp(&settings_path, session, port).await;
return;
}
Some(Command::Sessions) => {
print_sessions(&settings_path);
return;
}
None => {}
}
let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
let engine_tx = crate::engine::spawn(settings.clone(), crate::consts::ENGINE_CHANNEL_CAPACITY);
{
let max_logs = settings.lock().unwrap().max_logs;
let _ = engine_tx.tx.send(crate::engine::EngineMessage::SetMaxLogs(max_logs)).await;
}
{
let (active_profile_name, active_profile) = {
let mut settings_guard = settings.lock().unwrap();
settings_guard
.ensure_default_profile(&settings_path)
.expect("failed to ensure the Default profile exists");
let active_profile_name = settings_guard.active_profile.clone();
let active_profile = active_profile_name
.as_deref()
.and_then(|name| settings_guard.profile(name).cloned());
(active_profile_name, active_profile)
};
match (active_profile_name, active_profile) {
(Some(_), Some(active_profile)) => {
active_profile.load_into_engine(&engine_tx).await;
}
(Some(profile_name), None) => {
eprintln!(
"warning: active profile '{profile_name}' not found in {}; skipping",
settings_path.display()
);
}
(None, _) => {}
}
}
match args.file.clone() {
Some(path) => {
tokio::spawn(crate::engine::ingest::run_file(engine_tx.clone(), path));
}
None => {
tokio::spawn(crate::engine::ingest::run(engine_tx.clone()));
}
}
if let Some(metrics_url) = args.metrics_url.clone() {
tokio::spawn(crate::engine::ingest::run_metrics_poll(engine_tx.clone(), metrics_url));
}
let app = crate::api::router(engine_tx, settings, settings_path.clone());
let listener = TcpListener::bind((crate::consts::DEFAULT_BIND_HOST, args.port))
.await
.expect("failed to bind listener");
let bound_port = listener
.local_addr()
.expect("failed to read back bound address")
.port();
let source = match &args.file {
Some(path) => format!("tailing {}", path.display()),
None => "reading logs from stdin".to_string(),
};
let session_name = args.name.clone().unwrap_or_else(sessions::generate_name);
match sessions::register_session(&settings_path, &session_name, bound_port) {
Ok(_) => eprintln!(
"ottyl serving on http://localhost:{} ({}), session name: {}",
bound_port, source, session_name
),
Err(err) => eprintln!(
"ottyl serving on http://localhost:{} ({}); warning: failed to register session '{}': {}",
bound_port, source, session_name, err
),
}
axum::serve(listener, app)
.await
.expect("server error");
}
async fn run_mcp(settings_path: &std::path::Path, session: Option<String>, port: Option<u16>) {
use rmcp::ServiceExt;
let port = match (port, session) {
(Some(port), _) => port,
(None, Some(name)) => match sessions::resolve_session(settings_path, &name) {
Some(info) => info.port,
None => {
eprintln!("error: no running ottyl session named '{name}'");
std::process::exit(1);
}
},
(None, None) => {
eprintln!("error: `ottyl mcp` requires --session <name> or --port <port>");
std::process::exit(1);
}
};
let base_url = format!("http://localhost:{port}");
let service = crate::api::mcp_server::OttylMcp::new(base_url)
.serve(rmcp::transport::stdio())
.await
.expect("failed to start MCP server");
service.waiting().await.expect("MCP server error");
}
fn print_sessions(settings_path: &std::path::Path) {
let dir = sessions::sessions_dir(settings_path);
let sessions = sessions::list_sessions(&dir);
if sessions.is_empty() {
println!("no known ottyl sessions");
return;
}
for session in sessions {
println!("{}\tport={}\tpid={}", session.name, session.port, session.pid);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn settings_path_defaults_to_none() {
let args = Args::parse_from(["ottyl"]);
assert_eq!(args.settings_path, None);
}
#[test]
fn settings_path_flag_overrides_default() {
let args = Args::parse_from(["ottyl", "--settings-path", "/tmp/custom-settings.json"]);
assert_eq!(
args.settings_path,
Some(PathBuf::from("/tmp/custom-settings.json"))
);
}
}
src/sessions.rs → src/cli/sessions.rs
-30
View File
@@ -1,8 +1,3 @@
//! Named session registry (ticket 068): each running `ottyl` process registers itself under a
//! name by writing a small file to `<settings dir>/sessions/<name>.json`, so a second CLI
//! invocation (e.g. `ottyl mcp --session <name>`, ticket 067) can resolve a name to a port without
//! the user copying one around by hand.
use std::fs;
use std::io;
use std::path::{Path, PathBuf};
@@ -10,7 +5,6 @@ use std::process;
use serde::{Deserialize, Serialize};
/// On-disk shape of a `sessions/<name>.json` file.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SessionInfo {
pub name: String,
@@ -18,8 +12,6 @@ pub struct SessionInfo {
pub pid: u32,
}
/// The `sessions/` directory for a given settings file, following the same
/// sibling-directory convention as `profiles/` (see `settings::migrate_legacy_profiles_file`).
pub fn sessions_dir(settings_path: &Path) -> PathBuf {
settings_path
.parent()
@@ -31,10 +23,6 @@ fn session_file_path(dir: &Path, name: &str) -> PathBuf {
dir.join(format!("{name}.json"))
}
/// Best-effort liveness check for a PID. Shells out to `kill -0` on unix (no signal is delivered,
/// only permission/existence is checked); assumed alive on other platforms, which just means a
/// stale session file there waits for an explicit name collision to be overwritten instead of
/// being proactively cleaned up.
#[cfg(unix)]
fn is_pid_alive(pid: u32) -> bool {
process::Command::new("kill")
@@ -51,9 +39,6 @@ fn is_pid_alive(_pid: u32) -> bool {
true
}
/// Removes session files whose owning process is no longer running. Best-effort: there's no
/// shutdown hook to unregister a session on graceful exit, so this runs on every startup instead.
/// Any I/O error reading a single entry is ignored — a future call gets another chance.
pub fn cleanup_stale(dir: &Path) {
let Ok(entries) = fs::read_dir(dir) else {
return;
@@ -75,16 +60,10 @@ pub fn cleanup_stale(dir: &Path) {
}
}
/// Generates a random human-readable session name (e.g. `comic-gazelle`) for when `--name` isn't
/// given explicitly.
pub fn generate_name() -> String {
petname::petname(2, "-").unwrap_or_else(|| format!("session-{}", process::id()))
}
/// Registers this process as session `name` on `port`: cleans up stale entries, then writes
/// `sessions/<name>.json`, overwriting any existing file for that name (documented behavior — a
/// second instance started under the same explicit `--name` as a still-running one simply
/// replaces its registration; the older process keeps running, just without a discoverable entry).
pub fn register_session(settings_path: &Path, name: &str, port: u16) -> io::Result<PathBuf> {
let dir = sessions_dir(settings_path);
fs::create_dir_all(&dir)?;
@@ -99,9 +78,6 @@ pub fn register_session(settings_path: &Path, name: &str, port: u16) -> io::Resu
Ok(path)
}
/// Looks up a session by name. Returns `None` if no such file exists, it fails to parse, or its
/// process is no longer alive (a stale entry left by a process that didn't exit through the normal
/// next-`register_session` cleanup path).
pub fn resolve_session(settings_path: &Path, name: &str) -> Option<SessionInfo> {
let dir = sessions_dir(settings_path);
let contents = fs::read_to_string(session_file_path(&dir, name)).ok()?;
@@ -109,8 +85,6 @@ pub fn resolve_session(settings_path: &Path, name: &str) -> Option<SessionInfo>
is_pid_alive(info.pid).then_some(info)
}
/// Lists every session file under `dir`, live or stale (callers that only want live sessions
/// should filter on `is_pid_alive`-backed `resolve_session` instead). Used by `ottyl sessions`.
pub fn list_sessions(dir: &Path) -> Vec<SessionInfo> {
let Ok(entries) = fs::read_dir(dir) else {
return Vec::new();
@@ -157,8 +131,6 @@ mod tests {
let sessions_dir_path = sessions_dir(&settings_path);
fs::create_dir_all(&sessions_dir_path).unwrap();
// A PID essentially guaranteed not to be alive (PID 1 exists but a huge, unlikely-assigned
// PID stands in as "not alive" without depending on system PID-reuse specifics).
let stale = SessionInfo { name: "foo".to_string(), port: 1111, pid: 999_999 };
fs::write(
session_file_path(&sessions_dir_path, "foo"),
@@ -166,8 +138,6 @@ mod tests {
)
.unwrap();
// Registering a live session under the same name succeeds and the stale cleanup pass (run
// as part of registration) doesn't error out on the bogus pre-existing entry.
register_session(&settings_path, "foo", 5555).unwrap();
let info = resolve_session(&settings_path, "foo").unwrap();
assert_eq!(info.port, 5555);
src/settings.rs → src/config/mod.rs
+14 -99
View File
@@ -1,5 +1,4 @@
//! Process-wide settings model: colour scheme, CSS entries, and profiles (ticket 049 folded
//! profile storage inline here instead of a separate `profiles.json`), persisted as JSON.
pub mod profile;
use std::collections::hash_map::DefaultHasher;
use std::fmt;
@@ -13,18 +12,11 @@ use std::time::{SystemTime, UNIX_EPOCH};
use directories::ProjectDirs;
use serde::{Deserialize, Serialize};
use crate::alerts::AlertRule;
use crate::profile::Profile;
use crate::engine::alerts::AlertRule;
use crate::config::profile::Profile;
/// Shared, process-wide settings handle. Cheap to clone.
pub type SettingsHandle = Arc<Mutex<Settings>>;
/// User's preferred colour scheme for the webapp.
///
/// `Dark` accepts the legacy `"grey"` serde value (ticket 004: "grey" was a
/// near-duplicate of "dark" and has been removed as a selectable option) so a
/// settings file saved before that change still loads cleanly instead of
/// falling back to `System`.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")]
pub enum ColorScheme {
@@ -37,33 +29,22 @@ pub enum ColorScheme {
Yellow,
Pink,
Orange,
/// Brighter, light-surfaced counterpart to `Green` (ticket 048).
#[serde(rename = "light_green")]
LightGreen,
/// Brighter, light-surfaced counterpart to `Red` (ticket 056).
#[serde(rename = "light_red")]
LightRed,
/// Brighter, light-surfaced counterpart to `Blue` (ticket 056).
#[serde(rename = "light_blue")]
LightBlue,
/// Brighter, light-surfaced counterpart to `Yellow` (ticket 056).
#[serde(rename = "light_yellow")]
LightYellow,
/// Brighter, light-surfaced counterpart to `Pink` (ticket 056).
#[serde(rename = "light_pink")]
LightPink,
/// Brighter, light-surfaced counterpart to `Orange` (ticket 056).
#[serde(rename = "light_orange")]
LightOrange,
#[default]
System,
}
/// A single CSS entry: content ottyl stores itself ("imported") or a reference to an external
/// path/URL ("linked", not deletable via the API). Every entry is always a selectable
/// colour-scheme theme (ticket 020 removed the "use as theme"/always-on-overlay distinction);
/// an entry persisted before that change with a stored `is_theme: false` still loads fine (the
/// field is simply ignored) and becomes selectable like any other entry.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum CssEntry {
@@ -71,25 +52,15 @@ pub enum CssEntry {
Linked { path_or_url: String },
}
/// Legacy on-disk shape of the (pre-049) `Profiles` pointer struct: only names/active-profile/dir
/// pointers, with the actual profile content living in a separate `profiles.json`. Kept only so an
/// old `settings.json` still parses; see `ProfilesOnDisk` and `Settings::migrate_legacy_profiles_file`.
#[derive(Debug, Clone, Deserialize, Default)]
struct LegacyProfilesPointer {
// `names` is intentionally not captured: it was always redundant with the legacy
// `profiles.json`'s actual profile list, which is what migration reads instead.
#[serde(default)]
active_profile: Option<String>,
/// Directory (relative to the settings file's directory, or absolute) the legacy `profiles.json`
/// lived under. Every version of this codebase ever wrote `"profiles"` here (no API ever let a
/// user change it), so migration doesn't bother re-deriving a custom value from this field.
#[serde(default)]
#[allow(dead_code)]
profiles_dir: Option<String>,
}
/// On-disk shape of the `profiles` field: either the new inline `Vec<Profile>` (ticket 049) or the
/// old pointer-struct shape, so a `settings.json` written before this ticket still deserializes.
#[derive(Debug, Clone, Deserialize)]
#[serde(untagged)]
enum ProfilesOnDisk {
@@ -103,9 +74,6 @@ impl Default for ProfilesOnDisk {
}
}
/// Raw on-disk shape of `Settings`, used only to support deserializing an old `settings.json` whose
/// `profiles` field was the pointer struct and had no top-level `active_profile` — see `Settings`'s
/// `#[serde(from = ...)]`.
#[derive(Debug, Clone, Deserialize)]
struct SettingsOnDisk {
color_scheme: ColorScheme,
@@ -147,46 +115,35 @@ impl From<SettingsOnDisk> for Settings {
}
}
/// The single, process-wide settings instance. Only `PartialEq` (not `Eq`): `alert_rules` carries `f64` fields, which aren't `Eq`.
///
/// `profiles`/`active_profile` used to be a separate `Profiles` pointer struct referencing a
/// standalone `profiles.json` (see `crate::profile::ProfileStore`, removed by ticket 049); profile
/// data now lives inline here, as the single source of truth. `Deserialize` goes through
/// `SettingsOnDisk` so a `settings.json` written before this ticket still loads (see
/// `migrate_legacy_profiles_file`, which then imports the old `profiles.json`, if any, on `load`).
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
#[serde(from = "SettingsOnDisk")]
pub struct Settings {
pub color_scheme: ColorScheme,
pub css_entries: Vec<CssEntry>,
/// Id of the selected custom theme entry, if any; `None` means `color_scheme` alone determines the active palette.
#[serde(default)]
pub custom_theme_id: Option<String>,
/// Every known profile, inline (ticket 049).
pub profiles: Vec<Profile>,
/// Name of the currently active profile, if any.
pub active_profile: Option<String>,
/// Maximum number of ingested logs kept in memory; `None` means unlimited.
#[serde(default)]
pub max_logs: Option<usize>,
/// Alert rule definitions.
#[serde(default)]
pub alert_rules: Vec<AlertRule>,
/// Ntfy topic every alert rule delivers to; `None` until configured in Settings.
#[serde(default)]
pub ntfy_topic: Option<String>,
/// Ntfy server override; `None` means `https://ntfy.sh`.
#[serde(default)]
pub ntfy_server: Option<String>,
}
/// Computes the default settings file path (`<config_dir>/settings.json`) from OS-conventional config directories; `None` if none could be determined.
pub fn default_settings_path() -> Option<PathBuf> {
ProjectDirs::from("", "", "ottyl").map(|dirs| dirs.config_dir().join("settings.json"))
ProjectDirs::from(
crate::consts::APP_QUALIFIER,
crate::consts::APP_ORGANIZATION,
crate::consts::APP_NAME,
)
.map(|dirs| dirs.config_dir().join(crate::consts::SETTINGS_FILE_NAME))
}
impl Settings {
/// Loads settings from `path`, creating and persisting defaults if missing. Any I/O or parse error on an existing file is logged and treated as defaults, without overwriting the unreadable file. Also runs the one-time legacy `profiles.json` migration (see `migrate_legacy_profiles_file`).
pub fn load(path: &Path) -> Settings {
match fs::read_to_string(path) {
Ok(contents) => match serde_json::from_str::<Settings>(&contents) {
@@ -224,7 +181,6 @@ impl Settings {
}
}
/// Persists these settings to `path` as pretty-printed JSON, creating any missing parent directories first.
pub fn save(&self, path: &Path) -> io::Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
@@ -234,14 +190,6 @@ impl Settings {
fs::write(path, contents)
}
/// One-time migration (ticket 049): if `self.profiles` is empty (a fresh `Settings`, or one
/// parsed from a pre-049 `settings.json` whose profiles lived in a separate file) and a legacy
/// `<settings_path's parent>/profiles/profiles.json` exists, reads it in, merges it into
/// `self.profiles`, persists `self`, and — only once that save succeeds — renames the old file
/// (`.migrated` suffix) so it isn't re-imported on a later load. Every prior version of this
/// codebase always used the fixed `"profiles"` subdirectory name (no API ever let it be
/// reconfigured), so that's the only location checked. Best-effort: any failure is logged and
/// left for a future load to retry rather than panicking.
fn migrate_legacy_profiles_file(&mut self, settings_path: &Path) {
if !self.profiles.is_empty() {
return;
@@ -251,7 +199,7 @@ impl Settings {
.map(|parent| parent.join("profiles"))
.unwrap_or_else(|| PathBuf::from("profiles"))
.join("profiles.json");
let Some(profiles) = crate::profile::load_legacy_profile_store(&legacy_path) else {
let Some(profiles) = crate::config::profile::Profile::load_legacy_store(&legacy_path) else {
return;
};
self.profiles = profiles;
@@ -271,14 +219,8 @@ impl Settings {
}
}
/// Migrates any pre-037/pre-038 metric or param shapes still present in `self.profiles` into
/// their current form (see `crate::profile::migrate_legacy_profile_shapes`), persisting `self`
/// if anything changed. Runs on every load — previously done inside the now-removed
/// `ProfileStore::load` (ticket 049 folded profile storage into `settings.json`) — so a profile
/// saved by an older version of ottyl, whether loaded inline here or just merged in by
/// `migrate_legacy_profiles_file`, still gets migrated forward.
fn migrate_legacy_profile_shapes(&mut self, settings_path: &Path) {
if crate::profile::migrate_legacy_profile_shapes(&mut self.profiles)
if crate::config::profile::migrate_legacy_profile_shapes(&mut self.profiles)
&& let Err(err) = self.save(settings_path)
{
eprintln!(
@@ -289,14 +231,10 @@ impl Settings {
}
}
/// Error returned by CSS-entry mutations and lookups.
#[derive(Debug)]
pub enum CssError {
/// No entry with the given id exists.
NotFound,
/// The entry is a `Linked` entry, not `Imported` — ottyl doesn't own a linked entry's content, so there's nothing to delete or serve back.
NotImported,
/// Underlying I/O failure: writing/reading/removing the CSS file, or persisting `settings.json`.
Io(io::Error),
}
@@ -318,7 +256,6 @@ impl From<io::Error> for CssError {
}
}
/// Computes the id used to reference a `CssEntry` over the API: the stored id for `Imported`, or a hash of `path_or_url` for `Linked` (which has no stored id), so the frontend can key both the same way.
pub fn css_entry_id(entry: &CssEntry) -> String {
match entry {
CssEntry::Imported { id, .. } => id.clone(),
@@ -332,7 +269,6 @@ fn linked_entry_id(path_or_url: &str) -> String {
format!("linked-{:016x}", hasher.finish())
}
/// Generates an id for a newly-imported CSS entry by hashing filename, content, and time, avoiding a `uuid` dependency for what's otherwise an opaque string.
fn generate_imported_css_id(filename: &str, content: &str) -> String {
let mut hasher = DefaultHasher::new();
filename.hash(&mut hasher);
@@ -350,7 +286,6 @@ fn imported_css_file_path(css_dir: &Path, id: &str) -> PathBuf {
}
impl Settings {
/// Imports raw CSS `content`: writes it to `<css_dir>/<id>.css`, records a `CssEntry::Imported` in `css_entries`, and persists settings. Returns the generated id. `css_dir` is expected to be the settings file's parent directory.
pub fn import_css(
&mut self,
settings_path: &Path,
@@ -369,14 +304,12 @@ impl Settings {
Ok(id)
}
/// Registers a reference to an external CSS path/URL without copying its content, and persists settings.
pub fn link_css(&mut self, settings_path: &Path, path_or_url: String) -> Result<(), CssError> {
self.css_entries.push(CssEntry::Linked { path_or_url });
self.save(settings_path)?;
Ok(())
}
/// Removes an imported CSS entry identified by `id`: deletes its file under `css_dir`, drops it from `css_entries`, and persists settings. Fails with `NotFound` or `NotImported` (linked entries aren't deletable this way).
pub fn remove_css(
&mut self,
settings_path: &Path,
@@ -393,7 +326,6 @@ impl Settings {
Some(CssEntry::Imported { .. }) => {
let idx = index.unwrap();
self.css_entries.remove(idx);
// Best-effort: the settings entry is already removed even if the file is gone.
let _ = fs::remove_file(imported_css_file_path(css_dir, id));
self.save(settings_path)?;
Ok(())
@@ -401,7 +333,6 @@ impl Settings {
}
}
/// Reads back the raw content of an imported CSS entry identified by `id`. Fails with `NotFound` or `NotImported` (linked entries have no local content — the frontend fetches `path_or_url` directly).
pub fn css_content(&self, css_dir: &Path, id: &str) -> Result<String, CssError> {
match self.css_entries.iter().find(|entry| css_entry_id(entry) == id) {
None => Err(CssError::NotFound),
@@ -419,7 +350,6 @@ mod tests {
#[test]
fn default_settings_path_ends_with_settings_json() {
// Can't assert the exact OS path without touching real config dirs, so just check the shape.
if let Some(path) = default_settings_path() {
assert_eq!(path.file_name().unwrap(), "settings.json");
assert!(path.parent().is_some());
@@ -466,10 +396,6 @@ mod tests {
assert_eq!(settings.color_scheme, ColorScheme::Dark);
}
/// A `settings.json` written before ticket 049 (nested `Profiles` pointer struct, no top-level
/// `active_profile`) still parses: the pointer's `active_profile` carries over, and `profiles`
/// starts empty (the real content lived in a separate `profiles.json`, migrated separately by
/// `Settings::load`, not by plain deserialization).
#[test]
fn legacy_profiles_pointer_shape_deserializes_with_empty_inline_profiles() {
let settings: Settings = serde_json::from_str(
@@ -494,13 +420,13 @@ mod tests {
settings.css_entries.push(CssEntry::Linked {
path_or_url: "https://example.com/style.css".to_string(),
});
settings.profiles.push(crate::profile::Profile::new("default".to_string()));
settings.profiles.push(crate::config::profile::Profile::new("default".to_string()));
settings.active_profile = Some("default".to_string());
settings.alert_rules.push(crate::alerts::AlertRule {
settings.alert_rules.push(crate::engine::alerts::AlertRule {
id: "rule1".to_string(),
name: "error spike".to_string(),
enabled: true,
condition: crate::alerts::AlertCondition::Immediate {
condition: crate::engine::alerts::AlertCondition::Immediate {
query: r#"regex("error")"#.to_string(),
},
});
@@ -543,7 +469,6 @@ mod tests {
assert!(css_file.exists());
assert_eq!(fs::read_to_string(&css_file).unwrap(), "body { color: red; }");
// Drop and reload settings from disk (simulates a restart).
drop(settings);
let reloaded = Settings::load(&settings_path);
assert!(reloaded.css_entries.iter().any(|entry| matches!(
@@ -593,7 +518,6 @@ mod tests {
let result = settings.remove_css(&settings_path, &css_dir, &linked_id);
assert!(matches!(result, Err(CssError::NotImported)));
// Entry must still be there — rejection shouldn't silently drop it.
assert_eq!(settings.css_entries.len(), 1);
}
@@ -624,9 +548,6 @@ mod tests {
assert!(matches!(result, Err(CssError::NotImported)));
}
/// Acceptance criterion (ticket 049): a pre-existing legacy `profiles_dir/profiles.json` is
/// merged into `settings.json` on the next load, and the old file is renamed (not re-imported on
/// a subsequent load).
#[test]
fn load_migrates_legacy_profiles_file_into_inline_settings() {
let dir = tempfile::tempdir().unwrap();
@@ -639,8 +560,6 @@ mod tests {
r#"[{"name":"web","params":[],"metrics":[],"dashboards":[]}]"#,
)
.unwrap();
// No settings.json yet: simulates upgrading a pre-049 install that never wrote one either,
// or one that did but whose nested pointer already carried an active_profile name.
fs::write(
&settings_path,
r#"{"color_scheme":"system","css_entries":[],"custom_theme_id":null,"profiles":{"names":["web"],"active_profile":"web","profiles_dir":"profiles"},"max_logs":null,"alert_rules":[],"ntfy_topic":null,"ntfy_server":null}"#,
@@ -652,19 +571,15 @@ mod tests {
assert_eq!(settings.profiles[0].name, "web");
assert_eq!(settings.active_profile, Some("web".to_string()));
// The legacy file is gone (renamed), so it isn't re-imported.
assert!(!legacy_path.exists());
let migrated_path = legacy_path.with_extension("json.migrated");
assert!(migrated_path.exists());
// Reload from disk: the migrated settings.json now carries profiles inline directly.
let reloaded = Settings::load(&settings_path);
assert_eq!(reloaded.profiles.len(), 1);
assert_eq!(reloaded.profiles[0].name, "web");
}
/// A fresh install (no legacy file at all) never creates `profiles.json` — it's not even looked
/// for beyond the one migration check, and `profiles` simply stays empty.
#[test]
fn load_without_legacy_file_leaves_profiles_empty() {
let dir = tempfile::tempdir().unwrap();
src/profile.rs → src/config/profile.rs
+193 -386
View File
File diff suppressed because it is too large Load Diff
src/consts.rs
+54
View File
@@ -0,0 +1,54 @@
use std::time::Duration;
// mpsc channel capacity between ingest/API callers and the engine actor task.
pub const ENGINE_CHANNEL_CAPACITY: usize = 1024;
// Hard ceiling on how many params `auto_declare_params` will ever create automatically over the
// life of the engine, so one unusual input can't inflate the declared-param set forever.
pub const MAX_AUTO_DECLARED_PARAMS: usize = 64;
// Per-line ceiling on how many *new* params a single ingested line may contribute to the
// auto-derived set, so one unusually wide line can't burn the whole per-run budget at once.
pub const MAX_NEW_AUTO_PARAMS_PER_LINE: usize = 8;
// How long a sort helper index may sit unused before it's purged and left to rebuild on demand.
pub const SORT_HELPER_INDEX_PURGE_AFTER: Duration = Duration::from_secs(300);
// Window and capacity for the rolling ingestion-to-engine-processing lag high watermark.
pub const INGEST_LAG_WATERMARK_WINDOW_MS: i64 = 5000;
pub const INGEST_LAG_WATERMARK_MAX_ENTRIES: usize = 5;
// Bucket width used to merge multiple sub-series into one combined time series.
pub const MERGE_WINDOW_MS: i64 = 1_000;
// Default timeline query bucket width when the caller doesn't specify one.
pub const DEFAULT_TIMELINE_BUCKET_MS: i64 = 1_000;
// Reserved param names usable without declaring anything (ottyl-synthesized columns).
pub const OTTYL_RESERVED_PARAM_NAMES: [&str; 3] = ["ottyl_seq", "ottyl_ts", "ottyl_fmt"];
// Idle poll cadence while tailing stdin/a file for new log lines.
pub const STDIN_POLL_INTERVAL: Duration = Duration::from_millis(200);
// Poll cadence for a configured Prometheus/OpenMetrics `--metrics-url`.
pub const METRICS_POLL_INTERVAL: Duration = Duration::from_secs(15);
// Column-splitting regex fragments for the fixed-width logfmt column parser.
pub const COLUMN_SEPARATOR_PATTERN: &str = r"(?:\t|[ \t]{2,})";
pub const COLUMN_VALUE_PATTERN: &str = r"\S+(?: \S+)*";
// Name of the always-present profile every Settings file is seeded with.
pub const DEFAULT_PROFILE_NAME: &str = "Default";
// Name/label of the metric the Default profile is seeded with.
pub const SEEDED_METRIC_NAME: &str = "log_category";
pub const SEEDED_METRIC_LABEL: &str = "category";
// Host the HTTP server binds to; port is user-configurable, host is not.
pub const DEFAULT_BIND_HOST: &str = "0.0.0.0";
// App identity used to resolve the OS-conventional settings directory, and the settings file name.
pub const APP_QUALIFIER: &str = "";
pub const APP_ORGANIZATION: &str = "";
pub const APP_NAME: &str = "ottyl";
pub const SETTINGS_FILE_NAME: &str = "settings.json";
src/data_value.rs
-154
View File
@@ -1,154 +0,0 @@
use crate::metric_primitive::MetricPrimitive;
use crate::timestamp::Timestamp;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
/// A named, standalone parameter: how to pull a value out of a matching log line, and what kind it
/// parses as. Ticket 037 unified this with the metric-declaration-embedded matcher that used to
/// exist alongside it (`RegexToMetricPrimitiveMatcher`) — a metric now references params by name
/// (`MetricDeclaration::label_params`/`value_param`) instead of redeclaring its own extractor per
/// param, so a param declared once (e.g. via the Params panel) can be reused as a label by one
/// metric and as the value by another without re-entering its regex/capture-group/kind.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ParamDeclaration {
pub name: String,
pub message_match_regex: String,
pub capture_group: String,
/// Alternative extraction source to `capture_group`: a dot-separated JSON field path (e.g.
/// `foo.bar`), read from lines that parse as a JSON object instead of via the regex capture
/// group. When present and non-empty, this takes precedence over `capture_group`.
/// `#[serde(default)]` so profile files/requests saved before this field existed still load
/// as plain regex-capture-group params.
#[serde(default)]
pub json_field_path: Option<String>,
pub kind: MetricPrimitiveKind,
/// A third extraction source, alongside `capture_group`/`json_field_path`: when set, this
/// param's value is never parsed from text at all — it's computed as elapsed time since the
/// previous event that produced the same label set (ticket 038). Takes precedence over
/// `json_field_path`/`capture_group` when set, and `message_match_regex`/`capture_group`/
/// `json_field_path` are simply unused in that case (`declare_param` doesn't validate them for
/// this shape). Only meaningful paired with `kind == Duration`, since elapsed time is the only
/// primitive this computation can produce — `declare_param` enforces that pairing.
/// `#[serde(default)]` so params declared before this field existed keep parsing as plain
/// regex/JSON-field extraction.
#[serde(default)]
pub elapsed_since_last: bool,
/// Set when this param was discovered by `crate::engine::auto_declare_params` (a nom-based
/// `key=value` scan run on every ingested line) rather than declared through `POST /api/params`.
/// `declare_param` refuses to overwrite an auto-derived param by name, and it's never persisted
/// to a profile — see `crate::engine::declare_param` and `crate::api::reload_engine_declarations`.
/// `#[serde(default)]` so profiles/requests saved before this field existed still load as
/// ordinary (non-auto) params.
#[serde(default)]
pub auto_derived: bool,
}
/// Reads a value out of a JSON object by dot-separated field path (e.g. `"foo.bar"`), for
/// [`ParamDeclaration::json_field_path`]. Returns `None` when `text` doesn't parse as a JSON
/// object, the path doesn't resolve to a value, or the resolved value isn't a scalar
/// (string/number/bool) — there's no sensible flat text form of a nested object/array to feed into
/// `parse_metric_primitive`.
pub fn extract_json_field(text: &str, field_path: &str) -> Option<String> {
let root: serde_json::Value = serde_json::from_str(text).ok()?;
let mut current = root.as_object()?;
let mut segments = field_path.split('.').peekable();
loop {
let segment = segments.next()?;
let next_value = current.get(segment)?;
if segments.peek().is_none() {
return match next_value {
serde_json::Value::String(s) => Some(s.clone()),
serde_json::Value::Number(n) => Some(n.to_string()),
serde_json::Value::Bool(b) => Some(b.to_string()),
_ => None,
};
}
current = next_value.as_object()?;
}
}
/// Flattens a JSON object into dot-separated field paths mapped to each leaf scalar's raw text form
/// (string/number/bool), for [`crate::engine::auto_declare_params`] when a line is JSON
/// (`crate::logfmt::detect_format`) rather than logfmt — the JSON equivalent of
/// `crate::logfmt::extract_pairs`'s `key=value` scan. Nested objects recurse, dot-joining the path
/// (`"foo": {"bar": 1}` yields `"foo.bar"`); arrays and null are skipped since there's no single leaf
/// scalar to declare a param against, matching `extract_json_field`'s own restriction to scalars.
/// Returns an empty vec for anything that isn't a JSON object, same as `extract_pairs` returns empty
/// for a line with no `key=value` pairs.
pub fn flatten_json_object(text: &str) -> Vec<(String, String)> {
let Ok(serde_json::Value::Object(root)) = serde_json::from_str::<serde_json::Value>(text) else {
return Vec::new();
};
let mut results = Vec::new();
flatten_json_object_into(&root, "", &mut results);
results
}
fn flatten_json_object_into(
object: &serde_json::Map<String, serde_json::Value>,
prefix: &str,
results: &mut Vec<(String, String)>,
) {
for (key, value) in object {
let path = if prefix.is_empty() { key.clone() } else { format!("{prefix}.{key}") };
match value {
serde_json::Value::String(s) => results.push((path, s.clone())),
serde_json::Value::Number(n) => results.push((path, n.to_string())),
serde_json::Value::Bool(b) => results.push((path, b.to_string())),
serde_json::Value::Object(nested) => flatten_json_object_into(nested, &path, results),
serde_json::Value::Null | serde_json::Value::Array(_) => {}
}
}
}
/// `rename_all = "lowercase"` matches other `kind`-style enums, so JSON bodies use "string"/"number"/"timestamp".
#[allow(dead_code)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MetricPrimitiveKind {
String,
Number,
Timestamp,
/// A literal i64 millisecond count. Note this is unrelated to
/// `ParamDeclaration::elapsed_since_last`, which is a separate, independent extraction source —
/// a param using that source always parses/produces `Duration`, but not every `Duration`-kind
/// param is computed that way (one can also just parse a literal duration out of captured text,
/// like any other kind).
Duration,
/// A standalone param declared with this kind doesn't require (or use) a capture group: its value is
/// simply whether `message_match_regex` matches the line, case-insensitively or per the regex's own
/// flags — see `crate::engine::derive_log_params`. Unlike other kinds it's present as `true`/`false`
/// on every line rather than omitted when unmatched.
Boolean,
}
/// Value type paired with [`crate::index_key::IndexKey`] in the engine's primary index.
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize)]
pub enum DataValue {
/// Paired with `IndexKey::IngestedLog`.
LogLiteral { text: String },
/// Paired with `IndexKey::ParamDeclaration`.
ParamDeclaration(ParamDeclaration),
/// Paired with `IndexKey::MetricDeclaration`. `label_params`/`value_param` reference standalone
/// `ParamDeclaration`s by name (resolved against the engine's param store in
/// `crate::engine::apply_metric_to_log`) rather than embedding their own extractor.
MetricDeclaration {
message_match_regex: String,
label_params: Vec<String>,
/// At most one value param per metric; its primitive becomes the event's bucket value instead of a label.
value_param: Option<String>,
/// See `crate::engine::DeclareMetricRequest::matched_labels`.
matched_labels: Option<BTreeMap<String, String>>,
/// See `crate::engine::DeclareMetricRequest::unmatched_labels`.
unmatched_labels: Option<BTreeMap<String, String>>,
},
/// Paired with `IndexKey::Bucket`.
BucketValue {
values: Vec<(u64, Timestamp, MetricPrimitive)>,
},
/// Paired with `IndexKey::LastMetricEvent`.
LastEventTimestamp {
timestamp: Timestamp,
},
}
src/engine.rs
-4465
View File
File diff suppressed because it is too large Load Diff
src/alerts.rs → src/engine/alerts.rs
+28 -70
View File
@@ -1,16 +1,5 @@
//! Alert rules: definitions, trigger evaluation, and ntfy delivery.
//!
//! Alert queries are restricted to regex/ottyl_seq/ottyl_ts/ottyl_fmt/json[field] — a metric[param] comparison would require a full bucket rescan per line, breaking the proportional-cost guarantee. json[field] is fine here since, like regex/ottyl_seq/ottyl_ts/ottyl_fmt, it only inspects the line's own text/fields, no index lookup.
//!
//! `Threshold`/`Ratio` windows are maintained via `crate::rate::RateWindow` — the same windowed-rate
//! implementation PromQL's `rate()` uses (`crate::engine::evaluate_promql`) — rather than a
//! bespoke `VecDeque`+prune loop, so the two agree on what "windowed rate" means (ticket 040). A
//! window is also backfilled once, on first evaluation, from already-ingested logs within its own
//! trailing window (see `backfill_window`): a rule created mid-session (or a process that replays
//! existing log content, e.g. `--file` tailing) no longer starts with a silently empty window.
use std::collections::hash_map::DefaultHasher;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::collections::{HashMap, HashSet};
use std::fmt;
use std::hash::{Hash, Hasher};
use std::io;
@@ -20,14 +9,12 @@ use std::time::{SystemTime, UNIX_EPOCH};
use regex::Regex;
use serde::{Deserialize, Serialize};
use crate::data_value::DataValue;
use crate::engine::{self, apply_compare_op, eval_json_comparison};
use crate::filter_datetime;
use crate::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::index_key::IndexKey;
use crate::rate::RateWindow;
use crate::settings::Settings;
use crate::timestamp::Timestamp;
use crate::engine::{self, eval_json_comparison};
use crate::engine::filter_datetime;
use crate::engine::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::engine::rate::RateWindow;
use crate::config::Settings;
use crate::engine::timestamp::Timestamp;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct AlertRule {
@@ -41,19 +28,16 @@ pub struct AlertRule {
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum AlertCondition {
Immediate { query: String },
/// Fires when `query`'s match count within a rolling `window_ms` crosses `threshold` (via `op`).
Threshold {
query: String,
op: CompareOp,
threshold: f64,
window_ms: i64,
},
/// Seen values live only in memory — a restart forgets them, so previously-seen values can fire again; a deliberate simplicity tradeoff, not an oversight.
NewValue {
field_regex: String,
capture_group: String,
},
/// A zero `count_b` skips evaluation for that line rather than treating the ratio as infinite.
Ratio {
query_a: String,
query_b: String,
@@ -86,7 +70,6 @@ impl From<io::Error> for AlertError {
}
}
/// Hashes the name and current time rather than pulling in a `uuid` dependency.
fn generate_alert_rule_id(name: &str) -> String {
let mut hasher = DefaultHasher::new();
name.hash(&mut hasher);
@@ -145,13 +128,6 @@ impl Settings {
}
}
/// Per-rule in-memory evaluation state; lives only for the process's lifetime — not persisted
/// across a restart with no replayed log history. Each `RateWindow` is backfilled once (see
/// `backfill_window`) from whatever matching history is already in the index at first evaluation,
/// so — unlike before ticket 040 — a rule doesn't start blind to matches ingested before it was
/// declared, or before this evaluation loop last started, as long as that history is still in the
/// in-memory log index (e.g. `--file` tailing replays existing file content from the start on
/// every process start).
#[derive(Debug, Default)]
pub struct AlertRuntimeState {
threshold_window: RateWindow,
@@ -163,12 +139,8 @@ pub struct AlertRuntimeState {
seen_values: HashSet<String>,
}
/// Builds a `RateWindow` seeded from already-ingested logs matching `query`, bounded to
/// `window_ms` before `now` (not the full log history) so the cost stays proportional to the
/// window rather than everything ever ingested. `before_sequence` excludes the line currently
/// being evaluated, which the caller pushes separately if it matches.
fn backfill_window(
index: &BTreeMap<IndexKey, DataValue>,
snapshot: &engine::EngineSnapshot,
query: &str,
before_sequence: u64,
now: Timestamp,
@@ -176,7 +148,7 @@ fn backfill_window(
) -> RateWindow {
let since_ms = now.as_millis() - window_ms;
let matching_timestamps =
engine::logs_since(index, before_sequence, since_ms, now.as_millis()).filter_map(
snapshot.logs_since(before_sequence, since_ms, now.as_millis()).filter_map(
|(sequence, timestamp, text)| {
(eval_alert_query(query, sequence, timestamp, text) == Ok(true)).then_some(timestamp)
},
@@ -184,7 +156,6 @@ fn backfill_window(
RateWindow::backfill(matching_timestamps, now, window_ms)
}
/// Evaluates a FilterBar query against a single line's own fields only (metric[param] comparisons aren't supported — see module doc; json[field] is supported since it's also line-local).
fn eval_alert_query(query: &str, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
let expr = filter_parser::parse(query).map_err(|err| err.to_string())?;
eval_alert_expr(&expr, sequence, timestamp, text)
@@ -203,7 +174,7 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
.parse::<f64>()
.map_err(|_| format!("'ottyl_seq' comparisons require a numeric value, got {raw:?}"))?,
};
Ok(apply_compare_op((sequence as f64).total_cmp(&target), *op))
Ok(op.matches((sequence as f64).total_cmp(&target)))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_ts" => {
let target_ms = match value {
@@ -212,14 +183,14 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
.map_err(|err| err.to_string())?
.as_millis(),
};
Ok(apply_compare_op(timestamp.as_millis().cmp(&target_ms), *op))
Ok(op.matches(timestamp.as_millis().cmp(&target_ms)))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_fmt" => {
let target = match value {
FilterValue::Number(n) => n.to_string(),
FilterValue::Raw(raw) => raw.clone(),
};
Ok(apply_compare_op(crate::logfmt::detect_format(text).cmp(target.as_str()), *op))
Ok(op.matches(crate::engine::logfmt::detect_format(text).cmp(target.as_str())))
}
FilterExpr::Comparison { metric, .. } => Err(format!(
"alert queries only support regex()/ottyl_seq/ottyl_ts/ottyl_fmt (not metric comparisons like \
@@ -237,11 +208,10 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
}
}
/// A malformed query (bad regex, unsupported metric comparison) is logged and simply never matches, rather than panicking.
pub fn evaluate_alerts(
rules: &[AlertRule],
states: &mut HashMap<String, AlertRuntimeState>,
index: &BTreeMap<IndexKey, DataValue>,
snapshot: &engine::EngineSnapshot,
sequence: u64,
timestamp: Timestamp,
text: &str,
@@ -253,7 +223,7 @@ pub fn evaluate_alerts(
}
let state = states.entry(rule.id.clone()).or_default();
if let Some(message) =
evaluate_condition(&rule.condition, state, index, sequence, timestamp, text, &rule.name)
evaluate_condition(&rule.condition, state, snapshot, sequence, timestamp, text, &rule.name)
{
fired.push((rule.clone(), message));
}
@@ -264,7 +234,7 @@ pub fn evaluate_alerts(
fn evaluate_condition(
condition: &AlertCondition,
state: &mut AlertRuntimeState,
index: &BTreeMap<IndexKey, DataValue>,
snapshot: &engine::EngineSnapshot,
sequence: u64,
timestamp: Timestamp,
text: &str,
@@ -281,14 +251,14 @@ fn evaluate_condition(
},
AlertCondition::Threshold { query, op, threshold, window_ms } => {
if !state.threshold_backfilled {
state.threshold_window = backfill_window(index, query, sequence, timestamp, *window_ms);
state.threshold_window = backfill_window(snapshot, query, sequence, timestamp, *window_ms);
state.threshold_backfilled = true;
}
match eval_alert_query(query, sequence, timestamp, text) {
Ok(true) => {
state.threshold_window.push(timestamp, *window_ms);
let count = state.threshold_window.count() as f64;
apply_compare_op(count.total_cmp(threshold), *op).then(|| {
op.matches(count.total_cmp(threshold)).then(|| {
format!("{rule_name}: match count {count} {op:?} threshold {threshold} (window {window_ms}ms)")
})
}
@@ -328,11 +298,11 @@ fn evaluate_condition(
return None;
}
if !state.ratio_a_backfilled {
state.ratio_a_window = backfill_window(index, query_a, sequence, timestamp, *window_ms);
state.ratio_a_window = backfill_window(snapshot, query_a, sequence, timestamp, *window_ms);
state.ratio_a_backfilled = true;
}
if !state.ratio_b_backfilled {
state.ratio_b_window = backfill_window(index, query_b, sequence, timestamp, *window_ms);
state.ratio_b_window = backfill_window(snapshot, query_b, sequence, timestamp, *window_ms);
state.ratio_b_backfilled = true;
}
if matched_a {
@@ -344,11 +314,10 @@ fn evaluate_condition(
let count_a = state.ratio_a_window.count() as f64;
let count_b = state.ratio_b_window.count() as f64;
if count_b == 0.0 {
// Zero denominator: skip rather than treat the ratio as infinite.
return None;
}
let observed = count_a / count_b;
apply_compare_op(observed.total_cmp(ratio), *op).then(|| {
op.matches(observed.total_cmp(ratio)).then(|| {
format!(
"{rule_name}: ratio {observed:.3} ({count_a}/{count_b}) {op:?} {ratio} (window {window_ms}ms)"
)
@@ -357,9 +326,6 @@ fn evaluate_condition(
}
}
/// Delivers to the single, process-wide ntfy sink configured in Settings. Errors are only logged
/// (no retry queue); spawned off the ingest hot path so a slow/unreachable sink never blocks ingestion.
/// A silently no-op'd delivery (no topic configured) is expected — a rule can be defined before the sink is.
pub async fn deliver_for_rule(rule_name: String, message: String, topic: Option<String>, server: Option<String>) {
let Some(topic) = topic else {
return;
@@ -370,7 +336,6 @@ pub async fn deliver_for_rule(rule_name: String, message: String, topic: Option<
}
}
/// Standalone and `AlertRule`-free so a "send test notification" endpoint can call it directly.
pub async fn deliver_ntfy(server: &str, topic: &str, message: &str) -> Result<(), String> {
let url = format!("{}/{}", server.trim_end_matches('/'), topic);
let response = reqwest::Client::new()
@@ -464,7 +429,7 @@ mod tests {
fn immediate_fires_on_matching_line_only() {
let rules = vec![immediate_rule(r#"regex("error")"#)];
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let index = engine::EngineSnapshot::default();
let (seq, ts, text) = line("all good", 0, 0);
assert!(evaluate_alerts(&rules, &mut states, &index, seq, ts, text).is_empty());
@@ -480,7 +445,7 @@ mod tests {
let mut rule = immediate_rule(r#"regex("error")"#);
rule.enabled = false;
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let index = engine::EngineSnapshot::default();
let (seq, ts, text) = line("error!", 0, 0);
assert!(evaluate_alerts(&[rule], &mut states, &index, seq, ts, text).is_empty());
}
@@ -499,14 +464,13 @@ mod tests {
},
};
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let index = engine::EngineSnapshot::default();
for (seq, ms) in [(0u64, 0i64), (1, 1000)] {
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, seq, Timestamp::from_millis(ms), "error");
assert!(fired.is_empty(), "should not fire before threshold is crossed");
}
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 2, Timestamp::from_millis(2000), "error");
assert_eq!(fired.len(), 1);
// Stays quiet for a non-matching line even though the window is full.
let fired = evaluate_alerts(&[rule], &mut states, &index, 3, Timestamp::from_millis(2100), "fine");
assert!(fired.is_empty());
}
@@ -525,9 +489,8 @@ mod tests {
},
};
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let index = engine::EngineSnapshot::default();
assert!(evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "error").is_empty());
// Second match arrives well after the first has aged out of the window.
assert!(evaluate_alerts(&[rule], &mut states, &index, 1, Timestamp::from_millis(5_000), "error").is_empty());
}
@@ -543,7 +506,7 @@ mod tests {
},
};
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "agent=curl");
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 1, Timestamp::from_millis(1), "agent=curl");
@@ -567,16 +530,12 @@ mod tests {
},
};
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
// Only an "error" so far; no "request" yet, so denominator is 0 -> skip.
let index = engine::EngineSnapshot::default();
assert!(evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "error").is_empty());
// One request: ratio 1/1 = 1.0 >= 0.5 -> fires.
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 1, Timestamp::from_millis(1), "request");
assert_eq!(fired.len(), 1);
// A second request: ratio 1/2 = 0.5 still >= 0.5 -> fires again.
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 2, Timestamp::from_millis(2), "request");
assert_eq!(fired.len(), 1);
// A third request with no more errors: ratio drops to 1/3 -> no longer fires.
let fired = evaluate_alerts(&[rule], &mut states, &index, 3, Timestamp::from_millis(3), "request");
assert!(fired.is_empty());
}
@@ -585,7 +544,7 @@ mod tests {
fn ottyl_fmt_alert_query_matches_detected_format() {
let rule = immediate_rule("ottyl_fmt==json");
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(
&[rule.clone()],
@@ -605,8 +564,7 @@ mod tests {
fn metric_comparison_in_alert_query_is_rejected_not_panicking() {
let rule = immediate_rule("some_metric[param]==5");
let mut states = HashMap::new();
let index: BTreeMap<IndexKey, DataValue> = BTreeMap::new();
// Should log a warning and simply not fire, never panic.
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(&[rule], &mut states, &index, 0, Timestamp::from_millis(0), "anything");
assert!(fired.is_empty());
}
src/engine/data_value.rs
+96
View File
@@ -0,0 +1,96 @@
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::timestamp::Timestamp;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ParamDeclaration {
pub name: String,
pub message_match_regex: String,
pub capture_group: String,
#[serde(default)]
pub json_field_path: Option<String>,
pub kind: MetricPrimitiveKind,
#[serde(default)]
pub elapsed_since_last: bool,
#[serde(default)]
pub auto_derived: bool,
#[serde(default)]
pub log_stream_gated: bool,
}
pub fn extract_json_field(text: &str, field_path: &str) -> Option<String> {
let root: serde_json::Value = serde_json::from_str(text).ok()?;
let mut current = root.as_object()?;
let mut segments = field_path.split('.').peekable();
loop {
let segment = segments.next()?;
let next_value = current.get(segment)?;
if segments.peek().is_none() {
return match next_value {
serde_json::Value::String(s) => Some(s.clone()),
serde_json::Value::Number(n) => Some(n.to_string()),
serde_json::Value::Bool(b) => Some(b.to_string()),
_ => None,
};
}
current = next_value.as_object()?;
}
}
pub fn flatten_json_object(text: &str) -> Vec<(String, String)> {
let Ok(serde_json::Value::Object(root)) = serde_json::from_str::<serde_json::Value>(text) else {
return Vec::new();
};
let mut results = Vec::new();
flatten_json_object_into(&root, "", &mut results);
results
}
fn flatten_json_object_into(
object: &serde_json::Map<String, serde_json::Value>,
prefix: &str,
results: &mut Vec<(String, String)>,
) {
for (key, value) in object {
let path = if prefix.is_empty() { key.clone() } else { format!("{prefix}.{key}") };
match value {
serde_json::Value::String(s) => results.push((path, s.clone())),
serde_json::Value::Number(n) => results.push((path, n.to_string())),
serde_json::Value::Bool(b) => results.push((path, b.to_string())),
serde_json::Value::Object(nested) => flatten_json_object_into(nested, &path, results),
serde_json::Value::Null | serde_json::Value::Array(_) => {}
}
}
}
#[allow(dead_code)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MetricPrimitiveKind {
String,
Number,
Timestamp,
Duration,
Boolean,
}
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize)]
pub enum DataValue {
LogLiteral { text: String },
ParamDeclaration(ParamDeclaration),
MetricDeclaration {
message_match_regex: String,
label_params: Vec<String>,
value_param: Option<String>,
matched_labels: Option<BTreeMap<String, String>>,
unmatched_labels: Option<BTreeMap<String, String>>,
},
BucketValue {
values: Vec<(u64, Timestamp, MetricPrimitive)>,
},
LastEventTimestamp {
timestamp: Timestamp,
},
}
src/filter_datetime.rs → src/engine/filter_datetime.rs
+1 -19
View File
@@ -1,21 +1,12 @@
//! Datetime parsing for the filter query language's timestamp comparisons: relative
//! humantime, time-of-day (resolved against the UTC calendar day of `now`), or a
//! full date/datetime literal.
// `#![allow(dead_code)]` stays since most of this module's surface is still only
// exercised by its own unit tests below.
#![allow(dead_code)]
use chrono::{DateTime, NaiveDate, NaiveTime, TimeZone, Utc};
use crate::timestamp::Timestamp;
use crate::engine::timestamp::Timestamp;
/// Error returned when a datetime filter literal can't be resolved to a [`Timestamp`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum DateTimeParseError {
/// The input matched a supported format's shape but its value was invalid, e.g. `25:99`.
Invalid { input: String, reason: String },
/// The input didn't match any supported format at all.
Unparseable(String),
}
@@ -37,8 +28,6 @@ impl std::fmt::Display for DateTimeParseError {
impl std::error::Error for DateTimeParseError {}
/// Parses a filter datetime literal into an absolute [`Timestamp`]; `now` is a parameter
/// (rather than calling `Timestamp::now()` internally) so callers can pass a fixed value in tests.
pub fn parse_datetime(input: &str, now: Timestamp) -> Result<Timestamp, DateTimeParseError> {
let trimmed = input.trim();
if trimmed.is_empty() {
@@ -60,8 +49,6 @@ pub fn parse_datetime(input: &str, now: Timestamp) -> Result<Timestamp, DateTime
Err(DateTimeParseError::Unparseable(input.to_string()))
}
/// Tries to interpret `input` as a signed relative humantime duration (`5h`, `-3m`);
/// returns `None` if it doesn't look like one at all, so the caller can try other formats.
fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTimeParseError>> {
let (sign, rest) = match input.strip_prefix('-') {
Some(rest) => (-1i128, rest),
@@ -71,7 +58,6 @@ fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTim
},
};
// Require a digit followed by an alphabetic unit so date-like inputs (e.g. "2024-01-15") fall through to the date parser instead of erroring here as an invalid duration.
let looks_like_duration = rest
.chars()
.next()
@@ -97,9 +83,7 @@ fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTim
}
}
/// Tries to interpret `input` as a full date or RFC3339-ish datetime literal.
fn try_full_datetime(input: &str) -> Option<Result<Timestamp, DateTimeParseError>> {
// Only attempt this path for inputs shaped like a date, e.g. "2024-01-15".
let looks_like_date = input.len() >= 8 && input.as_bytes()[4] == b'-';
if !looks_like_date {
return None;
@@ -123,7 +107,6 @@ fn try_full_datetime(input: &str) -> Option<Result<Timestamp, DateTimeParseError
}
}
/// Tries to interpret `input` as a bare time-of-day, resolved against the UTC calendar day of `now`.
fn try_time_of_day(
input: &str,
now: Timestamp,
@@ -147,7 +130,6 @@ fn try_time_of_day(
let now_millis = now.as_millis();
let now_dt = Utc.timestamp_millis_opt(now_millis).single().unwrap_or_else(|| {
// Fall back to the epoch date; shouldn't happen since `Timestamp` is always a valid instant.
Utc.timestamp_millis_opt(0).single().expect("epoch is valid")
});
let today = now_dt.date_naive();
src/engine/filter_parser.pest
+30
View File
@@ -0,0 +1,30 @@
WHITESPACE = _{ " " | "\t" | "\n" | "\r" }
file = { SOI ~ or_expr ~ EOI }
or_expr = { and_expr ~ (or_kw ~ and_expr)* }
and_expr = { term ~ (and_kw ~ term)* }
term = _{ regex_expr | comparison | "(" ~ or_expr ~ ")" }
and_kw = @{ "and" ~ !(ASCII_ALPHANUMERIC | "_") }
or_kw = @{ "or" ~ !(ASCII_ALPHANUMERIC | "_") }
regex_expr = { "regex" ~ "(" ~ string_literal ~ ")" }
string_literal = ${ "\"" ~ inner_str ~ "\"" }
inner_str = @{ (escaped_char | !("\"" | "\\") ~ ANY)* }
escaped_char = @{ "\\" ~ ANY }
comparison = _{ json_comparison | metric_comparison }
json_comparison = { "json" ~ "[" ~ param ~ "]" ~ op ~ value }
metric_comparison = { metric ~ ("[" ~ param ~ "]")? ~ op ~ value }
metric = @{ ident }
param = @{ ident }
ident = @{ (ASCII_ALPHANUMERIC | "_")+ }
op = { "<=" | ">=" | "==" | "!=" | "<" | ">" }
value = @{ (!(WHITESPACE | ")") ~ ANY)+ }
src/filter_parser.rs → src/engine/filter_parser.rs
+1 -18
View File
@@ -1,5 +1,3 @@
//! FilterBar query language: grammar (see `src/filter_parser.pest`), AST and parser.
//! Evaluating the AST against real log data is left to `engine.rs`.
#![allow(dead_code)]
use pest::Parser;
@@ -8,23 +6,18 @@ use pest_derive::Parser as PestParser;
use serde::{Deserialize, Serialize};
#[derive(PestParser)]
#[grammar = "filter_parser.pest"]
#[grammar = "engine/filter_parser.pest"]
struct FilterGrammar;
/// The parsed AST for a FilterBar query.
#[derive(Debug, Clone, PartialEq)]
pub enum FilterExpr {
/// `regex("...")` — raw-text regex search over log lines.
Regex(String),
/// `metric[param] <op> value` — a single metric comparison.
Comparison {
metric: String,
param: Option<String>,
op: CompareOp,
value: FilterValue,
},
/// `json[field] <op> value` — a comparison against a field of the line's parsed JSON.
/// Lines that aren't valid JSON (or lack the field) simply don't match.
JsonComparison {
field: String,
op: CompareOp,
@@ -34,8 +27,6 @@ pub enum FilterExpr {
Or(Box<FilterExpr>, Box<FilterExpr>),
}
/// Comparison operators for metric comparisons; also reused by `crate::alerts::AlertCondition`
/// for threshold/ratio comparisons (hence `Serialize`/`Deserialize`).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum CompareOp {
@@ -47,15 +38,12 @@ pub enum CompareOp {
Ne,
}
/// The right-hand side of a comparison: values that parse as a number become
/// [`FilterValue::Number`], everything else is kept as an opaque [`FilterValue::Raw`] string.
#[derive(Debug, Clone, PartialEq)]
pub enum FilterValue {
Number(f64),
Raw(String),
}
/// A located, human-readable parse error.
#[derive(Debug, Clone, PartialEq)]
pub struct FilterParseError {
message: String,
@@ -79,12 +67,10 @@ impl std::error::Error for FilterParseError {}
impl From<pest::error::Error<Rule>> for FilterParseError {
fn from(err: pest::error::Error<Rule>) -> Self {
// pest's Display already renders a located message, so use it and add a short prefix.
FilterParseError::new(format!("failed to parse filter expression: {err}"))
}
}
/// Parse a FilterBar query string into a [`FilterExpr`] AST.
pub fn parse(input: &str) -> Result<FilterExpr, FilterParseError> {
let mut pairs = FilterGrammar::parse(Rule::file, input)?;
@@ -102,7 +88,6 @@ pub fn parse(input: &str) -> Result<FilterExpr, FilterParseError> {
fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
debug_assert_eq!(pair.as_rule(), Rule::or_expr);
// `or_kw` is atomic so it's not silent, showing up as a sibling pair between `and_expr` operands — skip it rather than assuming a fixed alternating structure.
let mut and_exprs = pair
.into_inner()
.filter(|p| p.as_rule() != Rule::or_kw);
@@ -119,7 +104,6 @@ fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
fn build_and_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
debug_assert_eq!(pair.as_rule(), Rule::and_expr);
// Same reasoning as `build_or_expr` for `and_kw`.
let mut terms = pair
.into_inner()
.filter(|p| p.as_rule() != Rule::and_kw);
@@ -237,7 +221,6 @@ fn parse_value(s: &str) -> FilterValue {
}
}
/// Unescape a regex string literal's inner contents (`\"` -> `"`, `\\` -> `\`, `\x` -> `x`).
fn unescape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut chars = s.chars();
src/engine/index_key.rs
+32
View File
@@ -0,0 +1,32 @@
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::timestamp::Timestamp;
use serde::Serialize;
use std::collections::BTreeMap;
#[allow(dead_code)]
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub enum IndexKey {
Start,
IngestedLog {
sequence: u64,
timestamp: Timestamp,
},
LogParamSplit,
ParamDeclaration {
name: String,
},
ParamMetricSplit,
MetricDeclaration {
name: String,
},
MetricBucketSplit,
Bucket {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
LastMetricEvent {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
End,
}
src/ingest.rs → src/engine/ingest.rs
+16 -38
View File
@@ -1,20 +1,13 @@
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::time::Duration;
use crate::engine::{EngineMessage, IngestedLine, MetricSample};
use crate::timestamp::Timestamp;
use crate::consts::{METRICS_POLL_INTERVAL, STDIN_POLL_INTERVAL as POLL_INTERVAL};
use crate::engine::{EngineHandle, EngineMessage, IngestedLine, MetricSample};
use crate::engine::timestamp::Timestamp;
use tokio::fs::File;
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::sync::mpsc;
const POLL_INTERVAL: Duration = Duration::from_millis(200);
const METRICS_POLL_INTERVAL: Duration = Duration::from_secs(15);
/// Reads stdin line by line, sequences and timestamps each line, and forwards it
/// to the engine task.
pub async fn run(engine: mpsc::Sender<EngineMessage>) {
pub async fn run(engine: EngineHandle) {
let stdin = tokio::io::stdin();
let mut lines = BufReader::new(stdin).lines();
let mut sequence: u64 = 0;
@@ -27,13 +20,12 @@ pub async fn run(engine: mpsc::Sender<EngineMessage>) {
};
sequence += 1;
if engine.send(EngineMessage::Ingest(line)).await.is_err() {
if engine.tx.send(EngineMessage::Ingest(line)).await.is_err() {
break;
}
}
}
/// Identifies a specific file instance on disk, so a recreated path can be told apart from the same file simply growing.
#[derive(PartialEq, Eq, Clone, Copy)]
struct FileId(#[cfg(unix)] u64, #[cfg(not(unix))] ());
@@ -56,10 +48,7 @@ async fn open(path: &Path) -> Option<(File, FileId)> {
Some((file, file_id(&meta)))
}
/// Tails `path` like `tail -f`, resuming from the start if the file is deleted/recreated or
/// truncated in place rather than exiting or getting stuck; `sequence` keeps incrementing
/// across a recreation since it's ottyl's own line counter, not tied to the source file.
pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
pub async fn run_file(engine: EngineHandle, path: PathBuf) {
let mut sequence: u64 = 0;
'reopen: loop {
@@ -78,7 +67,6 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
let mut chunk = String::new();
match reader.read_line(&mut chunk).await {
Ok(0) => {
// EOF: decide whether the file was truncated/recreated or just hasn't grown yet.
match tokio::fs::metadata(&path).await {
Ok(meta) => {
let new_id = file_id(&meta);
@@ -87,9 +75,7 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
}
id = new_id;
}
Err(_) => {
// Path gone (mid-recreation); wait for it to reappear and reopen fresh.
}
Err(_) => {}
}
tokio::time::sleep(POLL_INTERVAL).await;
}
@@ -109,11 +95,10 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
};
sequence += 1;
if engine.send(EngineMessage::Ingest(line)).await.is_err() {
if engine.tx.send(EngineMessage::Ingest(line)).await.is_err() {
return;
}
}
// else: partial line at EOF (write in progress) — keep buffering.
}
Err(_) => continue 'reopen,
}
@@ -121,10 +106,7 @@ pub async fn run_file(engine: mpsc::Sender<EngineMessage>, path: PathBuf) {
}
}
/// Polls `url` on a fixed interval for a Prometheus/OpenMetrics-format response, pushing every parsed
/// sample straight into the engine. Errors (unreachable endpoint, non-2xx response) are only logged —
/// polling keeps retrying on the next tick rather than exiting.
pub async fn run_metrics_poll(engine: mpsc::Sender<EngineMessage>, url: String) {
pub async fn run_metrics_poll(engine: EngineHandle, url: String) {
let client = reqwest::Client::new();
loop {
match client.get(&url).send().await {
@@ -133,7 +115,7 @@ pub async fn run_metrics_poll(engine: mpsc::Sender<EngineMessage>, url: String)
Ok(body) => {
let now = Timestamp::now();
for sample in parse_openmetrics(&body, now) {
if engine.send(EngineMessage::IngestMetricSample(sample)).await.is_err() {
if engine.tx.send(EngineMessage::IngestMetricSample(sample)).await.is_err() {
return;
}
}
@@ -148,10 +130,6 @@ pub async fn run_metrics_poll(engine: mpsc::Sender<EngineMessage>, url: String)
}
}
/// Parses the Prometheus text exposition / OpenMetrics format: one sample per non-comment,
/// non-empty line, as `metric_name{label="value",...} value [timestamp_ms]` (labels and the
/// timestamp are both optional). A line that doesn't parse is skipped rather than aborting the
/// whole scrape.
fn parse_openmetrics(body: &str, now: Timestamp) -> Vec<MetricSample> {
body.lines().filter_map(|line| parse_openmetrics_line(line, now)).collect()
}
@@ -185,8 +163,6 @@ fn parse_openmetrics_line(line: &str, now: Timestamp) -> Option<MetricSample> {
Some(MetricSample { metric_name, labels, value, timestamp })
}
/// Splits on commas without handling an escaped comma inside a quoted label value — fine for the
/// common case this targets, not a full OpenMetrics-grammar parser.
fn parse_openmetrics_labels(raw: &str) -> BTreeMap<String, String> {
raw.split(',')
.filter_map(|pair| pair.split_once('='))
@@ -238,7 +214,9 @@ mod openmetrics_tests {
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
use std::io::Write;
use crate::engine::test_handle;
use tokio::sync::mpsc;
async fn recv_text(rx: &mut mpsc::Receiver<EngineMessage>) -> String {
@@ -258,7 +236,7 @@ mod tests {
let path = dir.path().join("app.log");
std::fs::write(&path, "first\n").unwrap();
let (tx, mut rx) = mpsc::channel(16);
let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone()));
assert_eq!(recv_text(&mut rx).await, "first");
@@ -274,7 +252,7 @@ mod tests {
let path = dir.path().join("app.log");
std::fs::write(&path, "before-restart\n").unwrap();
let (tx, mut rx) = mpsc::channel(16);
let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone()));
assert_eq!(recv_text(&mut rx).await, "before-restart");
@@ -291,7 +269,7 @@ mod tests {
let path = dir.path().join("app.log");
std::fs::write(&path, "long line before truncation\n").unwrap();
let (tx, mut rx) = mpsc::channel(16);
let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone()));
assert_eq!(recv_text(&mut rx).await, "long line before truncation");
@@ -312,7 +290,7 @@ mod tests {
let path = dir.path().join("app.log");
std::fs::write(&path, "one\n").unwrap();
let (tx, mut rx) = mpsc::channel(16);
let (tx, mut rx) = test_handle(16);
tokio::spawn(run_file(tx, path.clone()));
let first = match tokio::time::timeout(Duration::from_secs(2), rx.recv())
src/engine/logfmt.rs
+524
View File
@@ -0,0 +1,524 @@
use nom::branch::alt;
use nom::bytes::complete::{escaped, tag, take_until, take_while1};
use nom::character::complete::{char, none_of, one_of};
use nom::error::{Error, ErrorKind};
use nom::sequence::delimited;
use nom::IResult;
use regex::Regex;
use std::sync::LazyLock;
use crate::consts::{COLUMN_SEPARATOR_PATTERN, COLUMN_VALUE_PATTERN};
fn key(input: &str) -> IResult<&str, &str> {
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '.' || c == '-')(input)
}
fn quoted_value(input: &str) -> IResult<&str, &str> {
delimited(
char('"'),
alt((escaped(none_of("\\\""), '\\', one_of("\"\\nt")), nom::combinator::success(""))),
char('"'),
)(input)
}
fn bareword_value(input: &str) -> IResult<&str, &str> {
take_while1(|c: char| !c.is_whitespace())(input)
}
fn looks_like_base64_padding_artifact(value: &str) -> bool {
value.starts_with('=')
}
fn pair(input: &str) -> IResult<&str, (&str, &str, bool)> {
let (input, k) = key(input)?;
let (input, _) = char('=')(input)?;
if let Ok((rest, v)) = quoted_value(input) {
Ok((rest, (k, v, true)))
} else {
let (rest, v) = bareword_value(input)?;
if looks_like_base64_padding_artifact(v) {
return Err(nom::Err::Error(Error::new(input, ErrorKind::Verify)));
}
Ok((rest, (k, v, false)))
}
}
pub fn extract_pairs(line: &str) -> Vec<(&str, &str, bool)> {
let mut results = Vec::new();
let mut rest = line;
while !rest.is_empty() {
match pair(rest) {
Ok((remaining, found)) => {
results.push(found);
rest = remaining;
}
Err(_) => {
let mut chars = rest.chars();
chars.next();
rest = chars.as_str();
}
}
}
results
}
static LOG_STREAM_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"^\d{4}-\d{2}-\d{2}\s+\S+\s+0x[0-9a-fA-F]+\s+\S+\s+0x[0-9a-fA-F]+\s+\d+\s+\d+\s+[^\s:]+:\s",
)
.expect("static log_stream regex must compile")
});
pub fn is_log_stream_line(line: &str) -> bool {
LOG_STREAM_RE.is_match(line)
}
fn bounded_column_value_pattern(width: usize) -> String {
if width <= 1 {
r"\S".to_string()
} else {
format!(r"\S(?:.{{0,{}}}\S)?", width - 2)
}
}
fn is_separator_run(run: &str) -> bool {
run.len() >= 2 || run.contains('\t')
}
static WHITESPACE_RUN: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"[ \t]+").expect("static regex must compile"));
fn split_columns_with_offsets(line: &str, max_columns: Option<usize>) -> Vec<(usize, &str)> {
let mut columns = Vec::new();
let mut start = 0usize;
let mut consumed = 0usize;
for m in WHITESPACE_RUN.find_iter(line) {
if max_columns.is_some_and(|max| consumed + 1 >= max) {
break;
}
let run = &line[m.start()..m.end()];
if is_separator_run(run) {
columns.push((start, &line[start..m.start()]));
start = m.end();
consumed += 1;
}
}
columns.push((start, &line[start..]));
columns
}
fn split_columns(line: &str, max_columns: Option<usize>) -> Vec<&str> {
split_columns_with_offsets(line, max_columns).into_iter().map(|(_, s)| s).collect()
}
static COLUMN_NAME: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[A-Za-z][A-Za-z0-9_]*$").expect("static regex must compile"));
#[derive(Clone)]
pub struct LogStreamColumn {
pub name: String,
start: usize,
}
#[derive(Clone)]
pub struct LogStreamHeader {
pub columns: Vec<LogStreamColumn>,
tab_delimited: bool,
}
fn char_slice(s: &str, start: usize, end: Option<usize>) -> &str {
if end.is_some_and(|end| end <= start) {
return "";
}
let mut indices = s.char_indices();
let byte_start = indices.nth(start).map(|(i, _)| i).unwrap_or(s.len());
let byte_end = match end {
Some(end) => s.char_indices().nth(end).map(|(i, _)| i).unwrap_or(s.len()),
None => s.len(),
};
&s[byte_start..byte_end.max(byte_start)]
}
impl LogStreamHeader {
pub fn detect(line: &str) -> Option<LogStreamHeader> {
if is_log_stream_line(line) {
return None;
}
let mut columns = Vec::new();
for (byte_start, slice) in split_columns_with_offsets(line, None) {
let trimmed = slice.trim();
if trimmed.is_empty() {
continue;
}
let leading_ws = slice.len() - slice.trim_start().len();
let start = line[..byte_start + leading_ws].chars().count();
columns.push(LogStreamColumn { name: trimmed.to_string(), start });
}
if columns.len() < 2 || !columns.iter().all(|c| COLUMN_NAME.is_match(&c.name)) {
return None;
}
let tab_delimited = WHITESPACE_RUN.find_iter(line).any(|m| m.as_str().contains('\t'));
Some(LogStreamHeader { columns, tab_delimited })
}
pub fn split_row<'a>(&self, line: &'a str) -> Vec<&'a str> {
if self.tab_delimited {
return split_columns(line, Some(self.columns.len() + 1));
}
let mut result = Vec::with_capacity(self.columns.len() + 1);
for (i, column) in self.columns.iter().enumerate() {
match self.columns.get(i + 1) {
Some(next) => result.push(char_slice(line, column.start, Some(next.start)).trim()),
None => {
let tail = char_slice(line, column.start, None);
let pieces = split_columns(tail, Some(2));
result.push(pieces[0].trim());
if let Some(message) = pieces.get(1) {
result.push(message);
}
}
}
}
result
}
pub fn column_pattern(&self, index: usize, capture_group: &str) -> String {
if self.tab_delimited {
let mut pattern = String::from("^");
for _ in 0..index {
pattern.push_str(COLUMN_VALUE_PATTERN);
pattern.push_str(COLUMN_SEPARATOR_PATTERN);
}
pattern.push_str(&format!("(?P<{capture_group}>{COLUMN_VALUE_PATTERN})"));
return pattern;
}
let start = self.columns[index].start;
let value_pattern = match self.columns.get(index + 1) {
Some(next) => bounded_column_value_pattern(next.start.saturating_sub(start)),
None => COLUMN_VALUE_PATTERN.to_string(),
};
format!("^.{{{start}}}(?P<{capture_group}>{value_pattern})")
}
pub fn message_pattern(&self, capture_group: &str) -> String {
if self.tab_delimited {
let mut pattern = String::from("^");
for _ in 0..self.columns.len() {
pattern.push_str(COLUMN_VALUE_PATTERN);
pattern.push_str(COLUMN_SEPARATOR_PATTERN);
}
pattern.push_str(&format!("(?P<{capture_group}>.*)$"));
return pattern;
}
let start = self.columns.last().map(|c| c.start).unwrap_or(0);
format!("^.{{{start}}}{COLUMN_VALUE_PATTERN}{COLUMN_SEPARATOR_PATTERN}(?P<{capture_group}>.*)$")
}
}
fn go_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str)> {
let (input, _) = tag("{\"time\":\"")(input)?;
let (input, time) = take_until("\"")(input)?;
let (input, _) = tag("\",\"level\":\"")(input)?;
let (input, level) = take_until("\"")(input)?;
let (input, _) = tag("\",\"msg\":\"")(input)?;
let (input, msg) = take_until("\"")(input)?;
let (input, _) = char('"')(input)?;
Ok((input, (time, level, msg)))
}
pub fn is_go_log_line(text: &str) -> bool {
match go_log_prefix(text) {
Ok((rest, _)) => rest.trim_end().ends_with('}'),
Err(_) => false,
}
}
fn py_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str, &str)> {
let (input, date) = take_while1(|c: char| c.is_ascii_digit() || c == '-')(input)?;
let (input, _) = char(' ')(input)?;
let (input, time) = take_while1(|c: char| c.is_ascii_digit() || c == ':' || c == ',')(input)?;
let (input, _) = char(' ')(input)?;
let (input, level) = take_while1(|c: char| c.is_ascii_uppercase())(input)?;
let (input, _) = char(' ')(input)?;
let (input, logger) =
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')(input)?;
let (input, _) = char(' ')(input)?;
Ok((input, (date, time, level, logger)))
}
pub fn is_py_log_line(text: &str) -> bool {
match py_log_prefix(text) {
Ok((rest, _)) => !extract_pairs(rest).is_empty(),
Err(_) => false,
}
}
pub fn detect_format(text: &str) -> &'static str {
if is_log_stream_line(text) {
"log_stream"
} else if is_go_log_line(text) {
"go-log"
} else if is_py_log_line(text) {
"py-log"
} else if serde_json::from_str::<serde_json::Value>(text).is_ok_and(|value| value.is_object()) {
"json"
} else if !extract_pairs(text).is_empty() {
"logfmt"
} else {
"none"
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_quoted_value() {
let pairs = extract_pairs(r#"msg="entity_moved""#);
assert_eq!(pairs, vec![("msg", "entity_moved", true)]);
}
#[test]
fn extracts_unquoted_bareword_value() {
let pairs = extract_pairs("x1=-4");
assert_eq!(pairs, vec![("x1", "-4", false)]);
}
#[test]
fn extracts_mixed_quoted_and_unquoted_pairs_in_one_line() {
let pairs = extract_pairs(r#"ts=1788648105773 msg="entity_moved" x1=-4 y1=34"#);
assert_eq!(
pairs,
vec![
("ts", "1788648105773", false),
("msg", "entity_moved", true),
("x1", "-4", false),
("y1", "34", false),
]
);
}
#[test]
fn returns_empty_for_lines_with_no_pairs() {
assert!(extract_pairs("just some plain prose here").is_empty());
assert!(extract_pairs(r#"{"foo": {"bar": 42}}"#).is_empty());
}
#[test]
fn does_not_panic_on_empty_value_at_end_of_line() {
assert_eq!(extract_pairs("key="), Vec::<(&str, &str, bool)>::new());
assert_eq!(extract_pairs(r#"key="""#), vec![("key", "", true)]);
}
#[test]
fn keeps_escaped_quote_inside_quoted_value() {
let pairs = extract_pairs(r#"msg="he said \"hi\"""#);
assert_eq!(pairs, vec![("msg", r#"he said \"hi\""#, true)]);
}
#[test]
fn detects_json_object_lines_as_json() {
assert_eq!(detect_format(r#"{"level":"error","code":500}"#), "json");
}
#[test]
fn detects_key_value_lines_as_logfmt() {
assert_eq!(detect_format(r#"ts=1788648105773 msg="entity_moved" x1=-4"#), "logfmt");
}
#[test]
fn detects_plain_prose_as_none() {
assert_eq!(detect_format("just some plain prose here"), "none");
}
#[test]
fn a_bare_json_array_or_scalar_is_not_treated_as_json() {
assert_eq!(detect_format("[1,2,3]"), "none");
assert_eq!(detect_format("42"), "none");
}
#[test]
fn rejects_base64_padding_fragment_as_a_pair() {
let pairs = extract_pairs(
"qname: <mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='>, qtype: A, client pid: 12260 (firefox)",
);
assert!(
pairs.iter().all(|(k, _, _)| *k != "rKTEtG7JmnThhoKPbZiVrA"),
"base64 padding fragment must not be extracted as a pair: {pairs:?}"
);
}
const LOG_STREAM_FIXTURE: &str = include_str!("../fixtures/log_stream_sample.txt");
#[test]
fn detects_log_stream_shaped_lines() {
let data_line = LOG_STREAM_FIXTURE.lines().nth(1).unwrap();
assert_eq!(detect_format(data_line), "log_stream");
assert!(is_log_stream_line(data_line));
}
#[test]
fn header_row_is_not_detected_as_log_stream() {
let header = LOG_STREAM_FIXTURE.lines().next().unwrap();
assert!(!is_log_stream_line(header));
}
fn column_names(header: &LogStreamHeader) -> Vec<&str> {
header.columns.iter().map(|c| c.name.as_str()).collect()
}
#[test]
fn detects_log_stream_header_columns_generically() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).expect("fixture header must be detected");
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type", "Activity", "PID", "TTL"]);
assert!(!header.tab_delimited);
}
#[test]
fn a_log_stream_data_line_is_never_detected_as_its_own_header() {
let data_line = LOG_STREAM_FIXTURE.lines().nth(1).unwrap();
assert!(LogStreamHeader::detect(data_line).is_none());
}
#[test]
fn plain_prose_is_not_detected_as_a_log_stream_header() {
assert!(LogStreamHeader::detect("just some plain prose here").is_none());
assert!(LogStreamHeader::detect("ts=1788648105773 msg=\"entity_moved\"").is_none());
}
#[test]
fn splits_log_stream_row_into_named_columns_plus_trailing_message() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let data_line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let row = header.split_row(data_line);
assert_eq!(row.len(), header.columns.len() + 1);
assert_eq!(row[0], "2026-09-06 01:48:54.518117+0100");
assert_eq!(row[1], "0x4dfed2");
assert_eq!(row[2], "Default");
assert_eq!(row[3], "0x0");
assert_eq!(row[4], "606");
assert_eq!(row[5], "0");
assert_eq!(row[6], "WindowServer: (HSTouchHIDService) something happened");
}
#[test]
fn a_single_tab_separates_columns_exactly_like_a_run_of_spaces() {
let header = LogStreamHeader::detect("Timestamp\tThread\tType").unwrap();
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type"]);
assert!(header.tab_delimited);
let spaced = header.split_row("2026-09-06 01:48:54\t0x4dfed2\t\tDefault trailing message");
let tabbed = header.split_row("2026-09-06 01:48:54\t0x4dfed2\tDefault trailing message");
assert_eq!(spaced[0], "2026-09-06 01:48:54");
assert_eq!(spaced[1], "0x4dfed2");
assert_eq!(spaced[2], "Default trailing message");
assert_eq!(tabbed[0], "2026-09-06 01:48:54");
assert_eq!(tabbed[1], "0x4dfed2");
assert_eq!(tabbed[2], "Default trailing message");
}
#[test]
fn column_and_message_patterns_capture_the_same_fields_split_log_stream_row_finds() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let type_re = Regex::new(&header.column_pattern(2, "Type")).unwrap();
assert_eq!(type_re.captures(line).unwrap().name("Type").unwrap().as_str(), "Default");
let pid_re = Regex::new(&header.column_pattern(4, "PID")).unwrap();
assert_eq!(pid_re.captures(line).unwrap().name("PID").unwrap().as_str(), "606");
let message_re = Regex::new(&header.message_pattern("message")).unwrap();
assert_eq!(
message_re.captures(line).unwrap().name("message").unwrap().as_str(),
"WindowServer: (HSTouchHIDService) something happened"
);
let timestamp_re = Regex::new(&header.column_pattern(0, "Timestamp")).unwrap();
assert_eq!(
timestamp_re.captures(line).unwrap().name("Timestamp").unwrap().as_str(),
"2026-09-06 01:48:54.518117+0100"
);
}
#[test]
fn is_log_stream_line_rejects_non_columnar_lines_that_a_fixed_offset_column_regex_would_still_match() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let timestamp_re = Regex::new(&header.column_pattern(0, "Timestamp")).unwrap();
let continuation_line = "so_gencnt: 4042707 so_state: 0x0";
assert!(timestamp_re.captures(continuation_line).is_some(), "sanity: regex alone still matches garbage");
assert!(!is_log_stream_line(continuation_line));
assert!(!is_log_stream_line(header_line));
assert!(!is_log_stream_line("}"));
}
#[test]
fn a_wifi_stat_dump_line_is_still_recognized_as_log_stream_despite_embedded_key_value_pairs() {
let line = LOG_STREAM_FIXTURE
.lines()
.find(|line| line.contains("txRTSFrm="))
.expect("fixture must contain the WiFi stat dump line");
assert_eq!(detect_format(line), "log_stream");
assert!(!extract_pairs(line).is_empty());
}
const GO_LOG_FIXTURE: &str = include_str!("../fixtures/go_log_sample.txt");
const PY_LOG_FIXTURE: &str = include_str!("../fixtures/py_log_sample.txt");
#[test]
fn detects_go_log_shaped_lines() {
for line in GO_LOG_FIXTURE.lines() {
assert!(is_go_log_line(line), "expected go-log line to match: {line}");
assert_eq!(detect_format(line), "go-log");
}
}
#[test]
fn detects_py_log_shaped_lines() {
for line in PY_LOG_FIXTURE.lines() {
assert!(is_py_log_line(line), "expected py-log line to match: {line}");
assert_eq!(detect_format(line), "py-log");
}
}
#[test]
fn a_generic_json_object_is_not_detected_as_go_log() {
let line = r#"{"level":"error","code":500}"#;
assert!(!is_go_log_line(line));
assert_eq!(detect_format(line), "json");
}
#[test]
fn a_generic_logfmt_line_is_not_detected_as_py_log() {
let line = r#"ts=1788648105773 msg="entity_moved" x1=-4"#;
assert!(!is_py_log_line(line));
assert_eq!(detect_format(line), "logfmt");
}
#[test]
fn go_log_prefix_extracts_time_level_and_msg() {
let (rest, (time, level, msg)) =
go_log_prefix(r#"{"time":"2026-09-06T10:35:34.499921+01:00","level":"WARN","msg":"eventC","labelC":1}"#)
.unwrap();
assert_eq!(time, "2026-09-06T10:35:34.499921+01:00");
assert_eq!(level, "WARN");
assert_eq!(msg, "eventC");
assert_eq!(rest, r#","labelC":1}"#);
}
#[test]
fn py_log_prefix_extracts_date_time_level_and_logger() {
let (rest, (date, time, level, logger)) =
py_log_prefix("2026-09-06 10:35:55,217 INFO delete-py-log event=eventC labelC=1").unwrap();
assert_eq!(date, "2026-09-06");
assert_eq!(time, "10:35:55,217");
assert_eq!(level, "INFO");
assert_eq!(logger, "delete-py-log");
assert_eq!(rest, "event=eventC labelC=1");
}
}
src/metric_primitive.rs → src/engine/metric_primitive.rs
+1 -7
View File
@@ -1,18 +1,15 @@
use crate::timestamp::Timestamp;
use crate::engine::timestamp::Timestamp;
use serde::Serialize;
use std::cmp::Ordering;
use std::fmt;
/// A simple, self-describing value used by metrics/labels without pulling in JSON.
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize)]
pub enum MetricPrimitive {
String(String),
Number(Number),
Timestamp(Timestamp),
/// Milliseconds elapsed; distinct from `Number` so it reads unambiguously in JSON/label form.
Duration(i64),
/// A `Boolean`-kind param's value: whether `message_match_regex` matched the line. See `MetricPrimitiveKind::Boolean`.
Boolean(bool),
}
@@ -51,7 +48,6 @@ impl Ord for MetricPrimitive {
}
}
/// Textual form used to compare label filters (query-string values) against a formatted `MetricPrimitive` rather than parsing them into one.
impl fmt::Display for MetricPrimitive {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
@@ -65,7 +61,6 @@ impl fmt::Display for MetricPrimitive {
}
}
/// Numbers are kept as a tagged int/float pair so exact integers stay exact.
#[allow(dead_code)]
#[derive(Debug, Clone, Copy, Serialize)]
pub enum Number {
@@ -74,7 +69,6 @@ pub enum Number {
}
impl Number {
/// Widens to `f64` for cross-kind numeric comparisons; `pub(crate)` since the filter evaluator in `engine.rs` needs it.
pub(crate) fn as_f64(&self) -> f64 {
match self {
Number::Int(v) => *v as f64,
src/engine/mod.rs
+4433
View File
File diff suppressed because it is too large Load Diff
src/promql.rs → src/engine/promql.rs
+51 -73
View File
@@ -1,21 +1,12 @@
//! Hand-rolled parser for a small PromQL subset: metric name + label matchers, an optional `[5m]` range, and a top-level `sum()`/`avg()`/`count()`/`rate()`/`histogram()`/`gauge()` wrapper.
//! Only exact-match (`=`) label matchers are supported; `rate()` requires a duration range selector, while `histogram(metric[width])` reuses that `[...]` slot for a bare bucket width instead.
//! `sum()`/`avg()` may additionally wrap a `rate(...)` call directly (e.g. `sum(rate(requests[5m]))`), and either may be followed by a `by (label, ...)` clause to preserve one series per distinct value of the named labels instead of collapsing to a single series.
//! `count()` is a plain vector selector wrapper only (no `rate()` nesting) — it counts occurrences per group regardless of the metric's value, which is the deliberate opt-in for occurrence counting that `sum()`/`avg()` no longer fall back to for a `String`-kind metric (see `crate::engine::evaluate_promql`).
use std::time::Duration;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AggOp {
Sum,
Avg,
/// Occurrence count per group, ignoring the metric's stored value entirely — the explicit,
/// opt-in-by-name alternative to summing a `String`-kind metric's coerced-to-1.0 values.
Count,
Rate,
/// Buckets counter values into fixed-width buckets; requires `bucket_width`.
Histogram,
/// The most-recent counter value per label combination.
Gauge,
}
@@ -30,41 +21,35 @@ pub struct PromqlQuery {
pub agg: Option<AggOp>,
pub metric_name: String,
pub matchers: Vec<LabelMatcher>,
/// Duration range selector, e.g. `[5m]`; only meaningful for `rate()`.
pub range: Option<Duration>,
/// Bare-number range selector, e.g. `[10]`; only meaningful for `histogram()`, where it's the bucket width.
pub bucket_width: Option<f64>,
/// Set when `agg` is `Sum`/`Avg` and it wraps an inner `rate(metric[range])` rather than a bare vector
/// selector, e.g. `sum(rate(requests[5m]))`. `range` above carries the inner `rate()`'s duration in that case.
pub inner_rate: bool,
/// Labels named in a trailing `by (...)` clause; empty means "collapse to a single series" (the historical
/// behavior of `sum()`/`avg()`). When non-empty, one series is kept per distinct combination of these labels.
pub by_labels: Vec<String>,
}
/// What a `[...]` range selector turned out to hold: a duration, or a bare width.
enum RangeSelector {
Duration(Duration),
BareWidth(f64),
}
/// Parses a PromQL query string, returning a human-readable error message on malformed input.
pub fn parse_query(input: &str) -> Result<PromqlQuery, String> {
let mut parser = Parser::new(input);
let query = parser.parse_query()?;
parser.skip_whitespace();
if !parser.is_at_end() {
return Err(format!("unexpected trailing input: '{}'", parser.remaining()));
impl PromqlQuery {
pub fn parse(input: &str) -> Result<PromqlQuery, String> {
let mut parser = Parser::new(input);
let query = parser.parse_query()?;
parser.skip_whitespace();
if !parser.is_at_end() {
return Err(format!("unexpected trailing input: '{}'", parser.remaining()));
}
if query.agg == Some(AggOp::Rate) && query.range.is_none() {
return Err("rate() requires a range selector, e.g. rate(my_metric[5m])".to_string());
}
if query.agg == Some(AggOp::Histogram) && query.bucket_width.is_none() {
return Err(
"histogram() requires a bucket width, e.g. histogram(my_metric[10])".to_string(),
);
}
Ok(query)
}
if query.agg == Some(AggOp::Rate) && query.range.is_none() {
return Err("rate() requires a range selector, e.g. rate(my_metric[5m])".to_string());
}
if query.agg == Some(AggOp::Histogram) && query.bucket_width.is_none() {
return Err(
"histogram() requires a bucket width, e.g. histogram(my_metric[10])".to_string(),
);
}
Ok(query)
}
struct Parser<'a> {
@@ -177,7 +162,6 @@ impl<'a> Parser<'a> {
Ok(matchers)
}
/// Parses `[<digits><unit>]` (a duration, e.g. `[5m]`) or a bare `[<digits>]` (a bucket width).
fn parse_range_selector(&mut self) -> Result<Option<RangeSelector>, String> {
self.skip_whitespace();
if self.peek() != Some('[') {
@@ -239,7 +223,6 @@ impl<'a> Parser<'a> {
})
}
/// Parses a trailing `by (label, ...)` clause, if present; returns an empty vec otherwise.
fn parse_by_clause(&mut self) -> Result<Vec<String>, String> {
self.skip_whitespace();
let checkpoint = self.pos;
@@ -289,14 +272,11 @@ impl<'a> Parser<'a> {
_ => None,
};
self.skip_whitespace();
// `sum by (label) (metric)`: a `by (...)` clause may also precede the aggregation's
// parenthesized argument, not just follow it.
let prefix_by_labels = if agg.is_some() { self.parse_by_clause()? } else { Vec::new() };
self.skip_whitespace();
if agg.is_some() && self.peek() == Some('(') {
self.expect_char('(')?;
self.skip_whitespace();
// `sum()`/`avg()` may wrap `rate(metric[range])` directly instead of a bare vector selector.
let mut inner = if matches!(agg, Some(AggOp::Sum) | Some(AggOp::Avg)) {
let inner_checkpoint = self.pos;
let took_inner_rate = self.parse_identifier().is_ok_and(|id| id == "rate");
@@ -338,7 +318,6 @@ impl<'a> Parser<'a> {
if prefix_by_labels.is_empty() { trailing_by_labels } else { prefix_by_labels };
Ok(inner)
} else {
// Rewind and parse as a plain vector selector, covering metric names that collide with an agg_op keyword.
self.pos = checkpoint;
self.parse_vector_selector()
}
@@ -351,7 +330,7 @@ mod tests {
#[test]
fn parses_bare_metric_name() {
let query = parse_query("requests").unwrap();
let query = PromqlQuery::parse("requests").unwrap();
assert_eq!(query.agg, None);
assert_eq!(query.metric_name, "requests");
assert!(query.matchers.is_empty());
@@ -360,7 +339,7 @@ mod tests {
#[test]
fn parses_metric_with_single_label_matcher() {
let query = parse_query(r#"requests{path="/a"}"#).unwrap();
let query = PromqlQuery::parse(r#"requests{path="/a"}"#).unwrap();
assert_eq!(query.metric_name, "requests");
assert_eq!(
query.matchers,
@@ -370,7 +349,7 @@ mod tests {
#[test]
fn parses_metric_with_multiple_label_matchers() {
let query = parse_query(r#"requests{path="/a", method="GET"}"#).unwrap();
let query = PromqlQuery::parse(r#"requests{path="/a", method="GET"}"#).unwrap();
assert_eq!(query.matchers.len(), 2);
assert_eq!(query.matchers[0].name, "path");
assert_eq!(query.matchers[1].name, "method");
@@ -378,68 +357,67 @@ mod tests {
#[test]
fn parses_range_selector() {
let query = parse_query("requests[5m]").unwrap();
let query = PromqlQuery::parse("requests[5m]").unwrap();
assert_eq!(query.range, Some(Duration::from_secs(300)));
}
#[test]
fn parses_label_matchers_and_range_together() {
let query = parse_query(r#"requests{path="/a"}[1h]"#).unwrap();
let query = PromqlQuery::parse(r#"requests{path="/a"}[1h]"#).unwrap();
assert_eq!(query.matchers.len(), 1);
assert_eq!(query.range, Some(Duration::from_secs(3600)));
}
#[test]
fn parses_sum_aggregation() {
let query = parse_query(r#"sum(requests{path="/a"})"#).unwrap();
let query = PromqlQuery::parse(r#"sum(requests{path="/a"})"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Sum));
assert_eq!(query.metric_name, "requests");
}
#[test]
fn parses_avg_aggregation() {
let query = parse_query("avg(requests)").unwrap();
let query = PromqlQuery::parse("avg(requests)").unwrap();
assert_eq!(query.agg, Some(AggOp::Avg));
}
#[test]
fn parses_rate_with_range() {
let query = parse_query("rate(requests[5m])").unwrap();
let query = PromqlQuery::parse("rate(requests[5m])").unwrap();
assert_eq!(query.agg, Some(AggOp::Rate));
assert_eq!(query.range, Some(Duration::from_secs(300)));
}
#[test]
fn rejects_rate_without_range() {
let err = parse_query("rate(requests)").unwrap_err();
let err = PromqlQuery::parse("rate(requests)").unwrap_err();
assert!(err.contains("range selector"));
}
#[test]
fn rejects_empty_input() {
assert!(parse_query("").is_err());
assert!(PromqlQuery::parse("").is_err());
}
#[test]
fn rejects_unclosed_brace() {
assert!(parse_query(r#"requests{path="/a""#).is_err());
assert!(PromqlQuery::parse(r#"requests{path="/a""#).is_err());
}
#[test]
fn rejects_unclosed_bracket() {
assert!(parse_query("requests[5m").is_err());
assert!(PromqlQuery::parse("requests[5m").is_err());
}
#[test]
fn rejects_unknown_aggregation() {
// `median` isn't a supported agg_op, so this parses as a bare metric name and trips on the trailing `(...)`.
let err = parse_query("median(requests)").unwrap_err();
let err = PromqlQuery::parse("median(requests)").unwrap_err();
assert!(err.contains("unexpected trailing input"));
}
#[test]
fn parses_count_aggregation() {
let query = parse_query("count(requests)").unwrap();
let query = PromqlQuery::parse("count(requests)").unwrap();
assert_eq!(query.agg, Some(AggOp::Count));
assert_eq!(query.metric_name, "requests");
assert!(!query.inner_rate);
@@ -447,37 +425,37 @@ mod tests {
#[test]
fn parses_count_with_by_clause() {
let query = parse_query("count(requests) by (path)").unwrap();
let query = PromqlQuery::parse("count(requests) by (path)").unwrap();
assert_eq!(query.agg, Some(AggOp::Count));
assert_eq!(query.by_labels, vec!["path".to_string()]);
}
#[test]
fn rejects_regex_label_matcher() {
let err = parse_query(r#"requests{path=~"/a"}"#).unwrap_err();
let err = PromqlQuery::parse(r#"requests{path=~"/a"}"#).unwrap_err();
assert!(err.contains("regex label matchers"));
}
#[test]
fn rejects_missing_metric_name() {
assert!(parse_query(r#"{path="/a"}"#).is_err());
assert!(PromqlQuery::parse(r#"{path="/a"}"#).is_err());
}
#[test]
fn rejects_unknown_duration_unit() {
let err = parse_query("requests[5x]").unwrap_err();
let err = PromqlQuery::parse("requests[5x]").unwrap_err();
assert!(err.contains("duration unit"));
}
#[test]
fn rejects_trailing_garbage() {
let err = parse_query("requests extra").unwrap_err();
let err = PromqlQuery::parse("requests extra").unwrap_err();
assert!(err.contains("unexpected trailing input"));
}
#[test]
fn parses_histogram_with_bare_width() {
let query = parse_query("histogram(latency_ms[10])").unwrap();
let query = PromqlQuery::parse("histogram(latency_ms[10])").unwrap();
assert_eq!(query.agg, Some(AggOp::Histogram));
assert_eq!(query.metric_name, "latency_ms");
assert_eq!(query.bucket_width, Some(10.0));
@@ -486,27 +464,27 @@ mod tests {
#[test]
fn rejects_histogram_without_width() {
let err = parse_query("histogram(latency_ms)").unwrap_err();
let err = PromqlQuery::parse("histogram(latency_ms)").unwrap_err();
assert!(err.contains("bucket width"));
}
#[test]
fn parses_gauge_with_no_range_selector() {
let query = parse_query(r#"gauge(connections{host="a"})"#).unwrap();
let query = PromqlQuery::parse(r#"gauge(connections{host="a"})"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Gauge));
assert_eq!(query.matchers, vec![LabelMatcher { name: "host".to_string(), value: "a".to_string() }]);
}
#[test]
fn rate_range_selector_is_still_a_duration_not_a_bare_width() {
let query = parse_query("rate(requests[5m])").unwrap();
let query = PromqlQuery::parse("rate(requests[5m])").unwrap();
assert_eq!(query.range, Some(Duration::from_secs(300)));
assert_eq!(query.bucket_width, None);
}
#[test]
fn parses_sum_of_rate() {
let query = parse_query("sum(rate(requests[1m]))").unwrap();
let query = PromqlQuery::parse("sum(rate(requests[1m]))").unwrap();
assert_eq!(query.agg, Some(AggOp::Sum));
assert!(query.inner_rate);
assert_eq!(query.metric_name, "requests");
@@ -516,7 +494,7 @@ mod tests {
#[test]
fn parses_avg_of_rate_with_label_matchers() {
let query = parse_query(r#"avg(rate(requests{path="/a"}[5m]))"#).unwrap();
let query = PromqlQuery::parse(r#"avg(rate(requests{path="/a"}[5m]))"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Avg));
assert!(query.inner_rate);
assert_eq!(query.matchers, vec![LabelMatcher { name: "path".to_string(), value: "/a".to_string() }]);
@@ -524,13 +502,13 @@ mod tests {
#[test]
fn rejects_sum_of_rate_without_range() {
let err = parse_query("sum(rate(requests))").unwrap_err();
let err = PromqlQuery::parse("sum(rate(requests))").unwrap_err();
assert!(err.contains("range selector"));
}
#[test]
fn parses_sum_with_by_clause() {
let query = parse_query("sum(requests) by (path)").unwrap();
let query = PromqlQuery::parse("sum(requests) by (path)").unwrap();
assert_eq!(query.agg, Some(AggOp::Sum));
assert!(!query.inner_rate);
assert_eq!(query.by_labels, vec!["path".to_string()]);
@@ -538,44 +516,44 @@ mod tests {
#[test]
fn parses_sum_of_rate_with_by_clause_multiple_labels() {
let query = parse_query("sum(rate(requests[1m])) by (path, method)").unwrap();
let query = PromqlQuery::parse("sum(rate(requests[1m])) by (path, method)").unwrap();
assert!(query.inner_rate);
assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]);
}
#[test]
fn sum_without_by_clause_has_empty_by_labels() {
let query = parse_query("sum(requests)").unwrap();
let query = PromqlQuery::parse("sum(requests)").unwrap();
assert!(query.by_labels.is_empty());
}
#[test]
fn rejects_empty_by_clause() {
let err = parse_query("sum(requests) by ()").unwrap_err();
let err = PromqlQuery::parse("sum(requests) by ()").unwrap_err();
assert!(err.contains("at least one label"));
}
#[test]
fn rejects_unterminated_by_clause() {
assert!(parse_query("sum(requests) by (path").is_err());
assert!(PromqlQuery::parse("sum(requests) by (path").is_err());
}
#[test]
fn parses_sum_with_prefix_by_clause() {
let query = parse_query("sum by (path) (requests)").unwrap();
let query = PromqlQuery::parse("sum by (path) (requests)").unwrap();
assert_eq!(query.agg, Some(AggOp::Sum));
assert_eq!(query.by_labels, vec!["path".to_string()]);
}
#[test]
fn parses_sum_with_prefix_by_clause_multiple_labels() {
let query = parse_query("sum by (path, method) (requests)").unwrap();
let query = PromqlQuery::parse("sum by (path, method) (requests)").unwrap();
assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]);
}
#[test]
fn rejects_by_clause_both_before_and_after_aggregation() {
let err = parse_query("sum by (path) (requests) by (method)").unwrap_err();
let err = PromqlQuery::parse("sum by (path) (requests) by (method)").unwrap_err();
assert!(err.contains("only appear once"));
}
}
src/rate.rs → src/engine/rate.rs
+1 -36
View File
@@ -1,24 +1,7 @@
//! Ticket 040: the single windowed-rate/count implementation in the codebase, shared by
//! PromQL's `rate()` (`crate::engine::evaluate_promql`, via the `windowed_rate_series` adapter
//! there) and alert `Threshold`/`Ratio` conditions (`crate::alerts`). Previously each computed
//! its own trailing window independently — PromQL scanning indexed bucket data, alerts scanning
//! raw log-line matches with a hand-rolled `VecDeque` — and could disagree for equivalent-looking
//! rules simply because the window boundary was implemented twice.
//!
//! `count_in_window` is the one place that decides what "falls in the trailing window" means;
//! everything else in this module is built on top of it, either by batch-rescanning a full point
//! series (`rate_at`/`windowed_rate_series`, fine for PromQL's historical/one-shot evaluation) or
//! incrementally (`RateWindow`, for alerts, which see one point at a time and need cost
//! proportional to the window rather than a full rescan per ingested line).
use std::collections::VecDeque;
use crate::timestamp::Timestamp;
use crate::engine::timestamp::Timestamp;
/// How many `points` fall in the trailing `(now - window_ms, now]` window. Points are occurrence
/// markers rather than a continuous reading, so only their timestamps matter — the boundary is
/// exclusive on the old end and inclusive of `now` itself, matching the original
/// `windowed_rate_series` behavior this replaces.
fn count_in_window(points: &[(Timestamp, f64)], now: Timestamp, window_ms: i64) -> usize {
let now_ms = now.as_millis();
let window_start = now_ms - window_ms;
@@ -31,33 +14,21 @@ fn count_in_window(points: &[(Timestamp, f64)], now: Timestamp, window_ms: i64)
.count()
}
/// Trailing-window occurrence rate (points per second) ending at `now`: how many `points` fall in
/// `(now - window_ms, now]`, divided by the window width in seconds.
pub fn rate_at(points: &[(Timestamp, f64)], now: Timestamp, window_ms: i64) -> f64 {
let window_seconds = window_ms as f64 / 1000.0;
count_in_window(points, now, window_ms) as f64 / window_seconds
}
/// Evaluates `rate_at` at every one of `points`' own timestamps, turning an occurrence series
/// into a windowed rate series — used by PromQL's `rate()` so it plots as a real line rather than
/// one trailing point.
pub fn windowed_rate_series(points: &[(Timestamp, f64)], window_ms: i64) -> Vec<(Timestamp, f64)> {
points.iter().map(|(ts, _)| (*ts, rate_at(points, *ts, window_ms))).collect()
}
/// Incrementally maintained trailing window: for a caller that sees points arrive one at a time
/// (alert `Threshold`/`Ratio` conditions, evaluated per ingested line) and needs cost proportional
/// to the window rather than `count_in_window`'s full rescan per call. Assumes points are pushed
/// in non-decreasing timestamp order, true for ingested log lines.
#[derive(Debug, Default, Clone)]
pub struct RateWindow {
timestamps: VecDeque<Timestamp>,
}
impl RateWindow {
/// Builds a window pre-populated from already-seen history (e.g. currently-ingested logs
/// matching an alert's query), so a freshly-created window starts with whatever of that
/// history already falls in the trailing window instead of empty.
pub fn backfill(timestamps: impl IntoIterator<Item = Timestamp>, now: Timestamp, window_ms: i64) -> Self {
let mut window = Self::default();
for ts in timestamps {
@@ -67,14 +38,11 @@ impl RateWindow {
window
}
/// Records one occurrence at `timestamp` and drops anything that has aged out of the
/// resulting trailing window.
pub fn push(&mut self, timestamp: Timestamp, window_ms: i64) {
self.timestamps.push_back(timestamp);
self.prune(timestamp, window_ms);
}
/// Same boundary as `count_in_window`: drops entries at or before `now - window_ms`.
fn prune(&mut self, now: Timestamp, window_ms: i64) {
let window_start = now.as_millis() - window_ms;
while let Some(&front) = self.timestamps.front() {
@@ -102,9 +70,7 @@ mod tests {
(Timestamp::from_millis(1000), 1.0),
(Timestamp::from_millis(2000), 1.0),
];
// (1000, 2000]: only the point at 2000.
assert_eq!(rate_at(&points, Timestamp::from_millis(2000), 1000), 1.0);
// (0, 2000]: points at 1000 and 2000 -> 2 points / 2s.
assert_eq!(rate_at(&points, Timestamp::from_millis(2000), 2000), 1.0);
}
@@ -136,7 +102,6 @@ mod tests {
#[test]
fn rate_window_backfill_seeds_from_history_within_the_window() {
let history = [0, 500, 2000, 2500].map(Timestamp::from_millis);
// now=2500ms, window=1000ms -> window (1500, 2500]: only 2000 and 2500 qualify.
let window = RateWindow::backfill(history, Timestamp::from_millis(2500), 1000);
assert_eq!(window.count(), 2);
}
src/timestamp.rs → src/engine/timestamp.rs
-1
View File
@@ -1,7 +1,6 @@
use serde::Serialize;
use std::time::{SystemTime, UNIX_EPOCH};
/// Wall-clock instant stored as nanoseconds since the Unix epoch (not `std::time::Instant`, which has no epoch and can't be serialized).
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub struct Timestamp(i64);
src/filter_parser.pest
-49
View File
@@ -1,49 +0,0 @@
// Grammar for the FilterBar query language.
//
// Supports:
// - regex("...") raw-text regex search over log lines
// - metric_name[param] <op> value metric comparisons
// - json[field] <op> value JSON field comparisons (line parsed as JSON)
// - ottyl_seq <op> value reserved metric name (no [param])
// - ottyl_ts <op> value reserved metric name (no [param])
// - ottyl_fmt <op> value reserved metric name (no [param])
// - `and` / `or` boolean combinators, with `and` binding tighter than `or`
// - parenthesized sub-expressions
WHITESPACE = _{ " " | "\t" | "\n" | "\r" }
file = { SOI ~ or_expr ~ EOI }
or_expr = { and_expr ~ (or_kw ~ and_expr)* }
and_expr = { term ~ (and_kw ~ term)* }
term = _{ regex_expr | comparison | "(" ~ or_expr ~ ")" }
// These must be atomic (`@`): a non-atomic rule would let pest's implicit
// whitespace insertion slip in between the keyword and its own boundary
// check below, which breaks the lookahead (it would see the first
// non-whitespace character *after* skipping the separating space, instead
// of the character immediately following the keyword).
and_kw = @{ "and" ~ !(ASCII_ALPHANUMERIC | "_") }
or_kw = @{ "or" ~ !(ASCII_ALPHANUMERIC | "_") }
regex_expr = { "regex" ~ "(" ~ string_literal ~ ")" }
string_literal = ${ "\"" ~ inner_str ~ "\"" }
inner_str = @{ (escaped_char | !("\"" | "\\") ~ ANY)* }
escaped_char = @{ "\\" ~ ANY }
comparison = _{ json_comparison | metric_comparison }
// `json[field] <op> value` must be tried before `metric_comparison` since "json" is
// otherwise a syntactically valid (if never declared) metric name.
json_comparison = { "json" ~ "[" ~ param ~ "]" ~ op ~ value }
metric_comparison = { metric ~ ("[" ~ param ~ "]")? ~ op ~ value }
metric = @{ ident }
param = @{ ident }
ident = @{ (ASCII_ALPHANUMERIC | "_")+ }
op = { "<=" | ">=" | "==" | "!=" | "<" | ">" }
value = @{ (!(WHITESPACE | ")") ~ ANY)+ }
src/index_key.rs
-39
View File
@@ -1,39 +0,0 @@
use crate::metric_primitive::MetricPrimitive;
use crate::timestamp::Timestamp;
use serde::Serialize;
use std::collections::BTreeMap;
/// Key type for the engine's primary index. Variant order matters: derived `Ord` compares the discriminant first, carving the map into contiguous ranges in declaration order (logs, params, metric declarations, buckets).
#[allow(dead_code)]
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub enum IndexKey {
/// Marker for the start of the whole range. Never inserted, only navigated to.
Start,
IngestedLog {
sequence: u64,
timestamp: Timestamp,
},
/// Marker between the log range and the param range.
LogParamSplit,
ParamDeclaration {
name: String,
},
/// Marker between the param range and the metric declaration range.
ParamMetricSplit,
MetricDeclaration {
name: String,
},
/// Marker between metric declarations and their buckets.
MetricBucketSplit,
Bucket {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
/// Internal bookkeeping tracking the last event timestamp per metric+label, so a `Duration` param's first occurrence (which pushes no `Bucket`) still lets the next occurrence compute elapsed time.
LastMetricEvent {
metric_name: String,
labels: BTreeMap<String, MetricPrimitive>,
},
/// Marker for the end of the whole range. Nothing sorts after this.
End,
}
src/logfmt.rs
-651
View File
@@ -1,651 +0,0 @@
//! Fast, tolerant `key=value`/`key="value"` tokenizer (logfmt-ish), used by
//! `crate::engine::auto_declare_params` to discover param names as lines are ingested. Scans
//! anywhere in the line rather than requiring the whole line to be logfmt, so it works on lines
//! with an arbitrary non-logfmt prefix/suffix (e.g. `ts=... msg="..." x1=-4 y1=34`).
use nom::branch::alt;
use nom::bytes::complete::{escaped, tag, take_until, take_while1};
use nom::character::complete::{char, none_of, one_of};
use nom::error::{Error, ErrorKind};
use nom::sequence::delimited;
use nom::IResult;
use regex::Regex;
use std::sync::LazyLock;
fn key(input: &str) -> IResult<&str, &str> {
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '.' || c == '-')(input)
}
fn quoted_value(input: &str) -> IResult<&str, &str> {
delimited(
char('"'),
alt((escaped(none_of("\\\""), '\\', one_of("\"\\nt")), nom::combinator::success(""))),
char('"'),
)(input)
}
fn bareword_value(input: &str) -> IResult<&str, &str> {
take_while1(|c: char| !c.is_whitespace())(input)
}
/// A bareword value starting with a bare `=` means the raw text actually had `key==...`/`key===...` —
/// never a real logfmt pair. This is exactly what `pair` would otherwise produce when it lands
/// mid-way through a base64/hash blob whose padding (`=`/`==`) happens to sit right after a run of
/// alnum chars that `key()` greedily consumed — e.g. the `huig_dump.txt` fixture's
/// `<mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='>,`: `key()` eats `rKTEtG7JmnThhoKPbZiVrA`, the first `=` is
/// consumed as the pair's `=`, leaving a bareword value of `='>,` (ticket 050). Rejected here rather
/// than post-filtered by a caller so `extract_pairs` never reports the false positive to anyone.
fn looks_like_base64_padding_artifact(value: &str) -> bool {
value.starts_with('=')
}
/// One `key=value` pair found on a line: the key, the raw captured value (unescaped as-is, quotes
/// stripped but any inner `\"`/`\\` left literal since callers only need it for numeric-kind
/// inference), and whether the value was quoted.
fn pair(input: &str) -> IResult<&str, (&str, &str, bool)> {
let (input, k) = key(input)?;
let (input, _) = char('=')(input)?;
if let Ok((rest, v)) = quoted_value(input) {
Ok((rest, (k, v, true)))
} else {
let (rest, v) = bareword_value(input)?;
if looks_like_base64_padding_artifact(v) {
return Err(nom::Err::Error(Error::new(input, ErrorKind::Verify)));
}
Ok((rest, (k, v, false)))
}
}
/// Finds every `key=value`/`key="value"` pair anywhere in `line`. A position that doesn't start a
/// pair is skipped one char at a time, so this never fails outright — a line with no pairs at all
/// (plain prose, a JSON blob) just returns an empty vec.
pub fn extract_pairs(line: &str) -> Vec<(&str, &str, bool)> {
let mut results = Vec::new();
let mut rest = line;
while !rest.is_empty() {
match pair(rest) {
Ok((remaining, found)) => {
results.push(found);
rest = remaining;
}
Err(_) => {
let mut chars = rest.chars();
chars.next();
rest = chars.as_str();
}
}
}
results
}
/// Fixed-column shape of macOS `log stream` output (ticket 050/054): a line starts with
/// `Timestamp Thread Type Activity PID TTL` (date, time+tz, a hex thread id, a bareword type, a hex
/// activity id, a decimal pid, a decimal ttl) followed by a free-text `process: message` tail — see
/// `huig_dump.txt`'s header row and body. Anchored at the very start of the line and requires the
/// hex/decimal shape of each column, so it won't accidentally fire on ordinary logfmt/JSON/prose that
/// merely resembles it in one column. Checked ahead of JSON/logfmt in `detect_format` since it's the
/// most specific of the three (an all-columns-plus-hex-ids match is a much stronger signal than an
/// incidental `key=value` substring), and is deliberately a *format label* rather than a dedicated
/// `ottyl_fmt` value change to JSON/logfmt detection — a `log stream` line's message tail can still
/// separately contain real `key=value` pairs (see the WiFi/radio stat lines in the fixture), which
/// `crate::engine::auto_declare_params` keeps discovering via `extract_pairs` regardless of this
/// line's `ottyl_fmt` label.
static LOG_STREAM_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"^\d{4}-\d{2}-\d{2}\s+\S+\s+0x[0-9a-fA-F]+\s+\S+\s+0x[0-9a-fA-F]+\s+\d+\s+\d+\s+[^\s:]+:\s",
)
.expect("static log_stream regex must compile")
});
pub fn is_log_stream_line(line: &str) -> bool {
LOG_STREAM_RE.is_match(line)
}
/// Regex fragment for one column separator under the generic whitespace-columnar rule ticket 059
/// asks for: a run of 2+ whitespace chars, or a lone tab, separates columns; a single plain space does
/// not (so a value that legitimately contains one -- e.g. `log stream`'s `date time+tz` timestamp --
/// stays intact within its own column). `is_separator_run` is the same rule applied at split time;
/// this is that rule re-expressed as a regex fragment for the trailing message field's
/// `message_match_regex`, which re-derives its value the same way whenever a caller queries by it
/// later (declarations don't cache extracted values -- see `crate::engine::extract_param_raw`).
const COLUMN_SEPARATOR_PATTERN: &str = r"(?:\t|[ \t]{2,})";
/// A column's value: one or more non-whitespace tokens joined by single spaces, with no upper bound
/// on length. Used for the trailing (unbounded-width) message field and the last named column, where
/// there's no next column's start position to bound against.
const COLUMN_VALUE_PATTERN: &str = r"\S+(?: \S+)*";
/// Same idea as `COLUMN_VALUE_PATTERN`, but bounded to a known column `width` (in chars) -- needed for
/// every column except the last, since two adjacent `log stream` values can legitimately be separated
/// by only a single space when the first one nearly fills its column's on-screen width (e.g. the
/// `Timestamp` column: its own `date time+tz` value already contains one internal space, so a
/// content-only pattern can't tell that boundary apart from the one-space gap before the next column
/// starts -- seed this from `LogStreamColumn::start` positions instead). Greedily matches up to
/// `width` chars while requiring the match to end on a non-whitespace char, which is what excludes the
/// column's own trailing padding spaces from the capture.
fn bounded_column_value_pattern(width: usize) -> String {
if width <= 1 {
r"\S".to_string()
} else {
format!(r"\S(?:.{{0,{}}}\S)?", width - 2)
}
}
fn is_separator_run(run: &str) -> bool {
run.len() >= 2 || run.contains('\t')
}
static WHITESPACE_RUN: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"[ \t]+").expect("static regex must compile"));
/// Splits `line` at whitespace runs that qualify as a column separator (`is_separator_run`), returning
/// each piece alongside its byte offset in `line`; a lone plain space never splits. Consumes at most
/// `max_columns.map(|n| n - 1)` separators -- the final piece returned is the untouched remainder of
/// the line, including any further whitespace runs inside it. That's what lets a trailing unlabelled
/// `log stream` message tail (whose own internal spacing can be arbitrary) come back as one field
/// instead of being chopped up by it.
fn split_columns_with_offsets(line: &str, max_columns: Option<usize>) -> Vec<(usize, &str)> {
let mut columns = Vec::new();
let mut start = 0usize;
let mut consumed = 0usize;
for m in WHITESPACE_RUN.find_iter(line) {
if max_columns.is_some_and(|max| consumed + 1 >= max) {
break;
}
let run = &line[m.start()..m.end()];
if is_separator_run(run) {
columns.push((start, &line[start..m.start()]));
start = m.end();
consumed += 1;
}
}
columns.push((start, &line[start..]));
columns
}
fn split_columns(line: &str, max_columns: Option<usize>) -> Vec<&str> {
split_columns_with_offsets(line, max_columns).into_iter().map(|(_, s)| s).collect()
}
/// A column-name identifier a generated `message_match_regex`'s named capture group can use as-is
/// (the regex crate requires `[A-Za-z_][0-9A-Za-z_]*`).
static COLUMN_NAME: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[A-Za-z][A-Za-z0-9_]*$").expect("static regex must compile"));
/// One column named by a detected `log stream` header (ticket 059). `start` is that column's position
/// in `char`s (not bytes -- see `char_slice`) within the header line -- meaningful only when
/// `LogStreamHeader::tab_delimited` is `false` (see that field's doc comment).
pub struct LogStreamColumn {
pub name: String,
start: usize,
}
/// A detected `log stream` header: its named columns, plus whether they were tab- or space-separated.
/// `log stream`'s real output right-pads each header name to a fixed on-screen width using runs of
/// 2+ spaces, and every data line's values land in that same fixed alignment regardless of how much
/// of that width they individually fill -- so for that shape, a data line's column boundaries are
/// best found at the header's own fixed character positions, not by re-splitting the data line's own
/// whitespace (see `split_log_stream_row`'s doc comment for why that alone isn't reliable there).
/// A tab-separated header carries no such alignment guarantee (values aren't padded to a shared
/// width), but a lone tab is itself an unambiguous separator, so that case instead re-splits each data
/// line directly.
pub struct LogStreamHeader {
pub columns: Vec<LogStreamColumn>,
tab_delimited: bool,
}
/// Byte-indexed `str` slicing doesn't work directly with `LogStreamColumn::start` (a *char* offset),
/// since `log stream` output is expected to stay ASCII in practice but callers shouldn't panic if a
/// message contains multi-byte chars before the slice point. `end` of `None` slices to end of string.
fn char_slice(s: &str, start: usize, end: Option<usize>) -> &str {
if end.is_some_and(|end| end <= start) {
return "";
}
let mut indices = s.char_indices();
let byte_start = indices.nth(start).map(|(i, _)| i).unwrap_or(s.len());
let byte_end = match end {
Some(end) => s.char_indices().nth(end).map(|(i, _)| i).unwrap_or(s.len()),
None => s.len(),
};
&s[byte_start..byte_end.max(byte_start)]
}
/// Recognizes a `log stream`-style header row generically (ticket 059), instead of assuming the fixed
/// `Timestamp Thread Type Activity PID TTL` names: a short run of bareword column names separated by
/// qualifying whitespace runs (`split_columns_with_offsets`) that is, critically, *not itself* a
/// `log stream` data line (`is_log_stream_line`) -- so a header is never mistaken for the data line
/// immediately following it. `crate::engine::auto_declare_log_stream_params` caches the result across
/// ingested lines, since `log stream` prints its header once, before any data line, and this only ever
/// sees one line at a time.
pub fn detect_log_stream_header(line: &str) -> Option<LogStreamHeader> {
if is_log_stream_line(line) {
return None;
}
let mut columns = Vec::new();
for (byte_start, slice) in split_columns_with_offsets(line, None) {
let trimmed = slice.trim();
if trimmed.is_empty() {
continue;
}
let leading_ws = slice.len() - slice.trim_start().len();
let start = line[..byte_start + leading_ws].chars().count();
columns.push(LogStreamColumn { name: trimmed.to_string(), start });
}
if columns.len() < 2 || !columns.iter().all(|c| COLUMN_NAME.is_match(&c.name)) {
return None;
}
let tab_delimited = WHITESPACE_RUN.find_iter(line).any(|m| m.as_str().contains('\t'));
Some(LogStreamHeader { columns, tab_delimited })
}
/// Splits a `log stream` data line into `header`'s named columns plus, when present, one trailing
/// unlabelled tail -- `crate::engine::auto_declare_log_stream_params` maps the first
/// `header.columns.len()` pieces onto the header's names and the optional last piece onto the
/// `message` param.
///
/// For a space-padded fixed-width header, every column but the last is sliced at the header's own
/// fixed character positions (`LogStreamColumn::start`) rather than by re-splitting the data line's
/// own whitespace, since two adjacent values can legitimately be joined by only a single space when
/// the first nearly fills its column's width (e.g. `log stream`'s `Timestamp` column, whose value
/// already contains one internal space). Only the last named column's end is whitespace-run-detected
/// (`split_columns`), since that's what distinguishes its own value from the free-text message tail
/// `log stream` appends after it.
///
/// For a tab-delimited header, there's no shared on-screen width to align against, but a lone tab is
/// itself unambiguous, so this instead re-splits the data line directly on qualifying separators.
pub fn split_log_stream_row<'a>(line: &'a str, header: &LogStreamHeader) -> Vec<&'a str> {
if header.tab_delimited {
return split_columns(line, Some(header.columns.len() + 1));
}
let mut result = Vec::with_capacity(header.columns.len() + 1);
for (i, column) in header.columns.iter().enumerate() {
match header.columns.get(i + 1) {
Some(next) => result.push(char_slice(line, column.start, Some(next.start)).trim()),
None => {
let tail = char_slice(line, column.start, None);
let pieces = split_columns(tail, Some(2));
result.push(pieces[0].trim());
if let Some(message) = pieces.get(1) {
result.push(message);
}
}
}
}
result
}
/// Builds the `message_match_regex` for the `index`-th (0-based) of `header`'s named columns.
///
/// Fixed-width case: skips exactly `header.columns[index].start` chars, then captures this column's
/// own value bounded to its known width (`bounded_column_value_pattern`) -- unbounded for the last
/// column, which has no next column to bound against.
///
/// Tab-delimited case: skips `index` preceding columns' own value+separator pairs instead, since
/// there's no shared fixed width to anchor a character-position skip against.
///
/// Anchored at the start of the line since a `log stream` line never carries this fixed-per-run
/// column prefix anywhere but the very start.
pub fn log_stream_column_pattern(header: &LogStreamHeader, index: usize, capture_group: &str) -> String {
if header.tab_delimited {
let mut pattern = String::from("^");
for _ in 0..index {
pattern.push_str(COLUMN_VALUE_PATTERN);
pattern.push_str(COLUMN_SEPARATOR_PATTERN);
}
pattern.push_str(&format!("(?P<{capture_group}>{COLUMN_VALUE_PATTERN})"));
return pattern;
}
let start = header.columns[index].start;
let value_pattern = match header.columns.get(index + 1) {
Some(next) => bounded_column_value_pattern(next.start.saturating_sub(start)),
None => COLUMN_VALUE_PATTERN.to_string(),
};
format!("^.{{{start}}}(?P<{capture_group}>{value_pattern})")
}
/// Builds the `message_match_regex` for the trailing unlabelled tail after `header`'s last named
/// column: everything from the end of that column's own separator to end of line, verbatim -- its own
/// internal whitespace, however irregular, is never treated as a further separator.
pub fn log_stream_message_pattern(header: &LogStreamHeader, capture_group: &str) -> String {
if header.tab_delimited {
let mut pattern = String::from("^");
for _ in 0..header.columns.len() {
pattern.push_str(COLUMN_VALUE_PATTERN);
pattern.push_str(COLUMN_SEPARATOR_PATTERN);
}
pattern.push_str(&format!("(?P<{capture_group}>.*)$"));
return pattern;
}
let start = header.columns.last().map(|c| c.start).unwrap_or(0);
format!("^.{{{start}}}{COLUMN_VALUE_PATTERN}{COLUMN_SEPARATOR_PATTERN}(?P<{capture_group}>.*)$")
}
/// Go `log/slog` `NewJSONHandler` output (the `go-log` label): a single-line JSON object whose first
/// three fields are always `time`, `level`, `msg` in that fixed order (slog's `JSONHandler` writes the
/// built-in attrs before any caller-supplied ones), followed by zero or more arbitrary attrs and a
/// closing `}`. Parsed with nom rather than a full JSON decode since only that fixed prefix shape
/// needs verifying; the variable tail is deliberately not parsed field-by-field the way
/// `crate::engine::auto_declare_params` will separately pick up JSON leaf fields from the line.
/// Checked ahead of the generic JSON object check in `detect_format` since it's the more specific of
/// the two (see that function's doc comment for the general ordering rationale).
fn go_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str)> {
let (input, _) = tag("{\"time\":\"")(input)?;
let (input, time) = take_until("\"")(input)?;
let (input, _) = tag("\",\"level\":\"")(input)?;
let (input, level) = take_until("\"")(input)?;
let (input, _) = tag("\",\"msg\":\"")(input)?;
let (input, msg) = take_until("\"")(input)?;
let (input, _) = char('"')(input)?;
Ok((input, (time, level, msg)))
}
pub fn is_go_log_line(text: &str) -> bool {
match go_log_prefix(text) {
Ok((rest, _)) => rest.trim_end().ends_with('}'),
Err(_) => false,
}
}
/// Python `logging` module output in the shape `"%(asctime)s %(levelname)s %(name)s %(message)s"`
/// (the `py-log` label): a `YYYY-MM-DD HH:MM:SS,mmm` timestamp, an uppercase level word, a logger
/// name, then a message tail that (for `delete-py-log`, and any caller using `logger.info("event=%s
/// %s=%s", ...)`-style calls) is itself logfmt `key=value` pairs. Only the fixed prefix is parsed with
/// nom; the tail is handed to `extract_pairs` both to confirm the shape (a py-log line with no pairs
/// at all wouldn't be distinguishable from plain prose after this prefix) and because that's the same
/// extraction `crate::engine::auto_declare_params` needs anyway. Checked ahead of the generic logfmt
/// check in `detect_format` since it's the more specific of the two.
fn py_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str, &str)> {
let (input, date) = take_while1(|c: char| c.is_ascii_digit() || c == '-')(input)?;
let (input, _) = char(' ')(input)?;
let (input, time) = take_while1(|c: char| c.is_ascii_digit() || c == ':' || c == ',')(input)?;
let (input, _) = char(' ')(input)?;
let (input, level) = take_while1(|c: char| c.is_ascii_uppercase())(input)?;
let (input, _) = char(' ')(input)?;
let (input, logger) =
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')(input)?;
let (input, _) = char(' ')(input)?;
Ok((input, (date, time, level, logger)))
}
pub fn is_py_log_line(text: &str) -> bool {
match py_log_prefix(text) {
Ok((rest, _)) => !extract_pairs(rest).is_empty(),
Err(_) => false,
}
}
/// The value of the `ottyl_fmt` reserved param (`crate::engine::compile_filter`/`eval_alert_expr`):
/// which format a line was recognized as when parsed. Checked most-specific-first: `log stream`'s
/// fixed-column shape (see `LOG_STREAM_RE`'s doc comment for why it's the most specific), then the two
/// named-tool shapes `go-log` (a `time`/`level`/`msg`-prefixed JSON object) and `py-log` (a
/// timestamp/level/logger prefix over logfmt pairs) — each is a strict subset of the generic JSON/logfmt
/// case checked right after it, so they'd otherwise never be reached. JSON is checked next since a JSON
/// object can incidentally contain `key=value`-shaped substrings that `extract_pairs` would otherwise
/// pick up; only a top-level object counts (arrays/scalars aren't what `crate::engine::eval_json_comparison`
/// treats as JSON either). Anything else falls back to logfmt-style pair extraction, and a line with
/// none of the above is `"none"`.
pub fn detect_format(text: &str) -> &'static str {
if is_log_stream_line(text) {
"log_stream"
} else if is_go_log_line(text) {
"go-log"
} else if is_py_log_line(text) {
"py-log"
} else if serde_json::from_str::<serde_json::Value>(text).is_ok_and(|value| value.is_object()) {
"json"
} else if !extract_pairs(text).is_empty() {
"logfmt"
} else {
"none"
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_quoted_value() {
let pairs = extract_pairs(r#"msg="entity_moved""#);
assert_eq!(pairs, vec![("msg", "entity_moved", true)]);
}
#[test]
fn extracts_unquoted_bareword_value() {
let pairs = extract_pairs("x1=-4");
assert_eq!(pairs, vec![("x1", "-4", false)]);
}
#[test]
fn extracts_mixed_quoted_and_unquoted_pairs_in_one_line() {
let pairs = extract_pairs(r#"ts=1788648105773 msg="entity_moved" x1=-4 y1=34"#);
assert_eq!(
pairs,
vec![
("ts", "1788648105773", false),
("msg", "entity_moved", true),
("x1", "-4", false),
("y1", "34", false),
]
);
}
#[test]
fn returns_empty_for_lines_with_no_pairs() {
assert!(extract_pairs("just some plain prose here").is_empty());
assert!(extract_pairs(r#"{"foo": {"bar": 42}}"#).is_empty());
}
#[test]
fn does_not_panic_on_empty_value_at_end_of_line() {
assert_eq!(extract_pairs("key="), Vec::<(&str, &str, bool)>::new());
assert_eq!(extract_pairs(r#"key="""#), vec![("key", "", true)]);
}
#[test]
fn keeps_escaped_quote_inside_quoted_value() {
let pairs = extract_pairs(r#"msg="he said \"hi\"""#);
assert_eq!(pairs, vec![("msg", r#"he said \"hi\""#, true)]);
}
#[test]
fn detects_json_object_lines_as_json() {
assert_eq!(detect_format(r#"{"level":"error","code":500}"#), "json");
}
#[test]
fn detects_key_value_lines_as_logfmt() {
assert_eq!(detect_format(r#"ts=1788648105773 msg="entity_moved" x1=-4"#), "logfmt");
}
#[test]
fn detects_plain_prose_as_none() {
assert_eq!(detect_format("just some plain prose here"), "none");
}
#[test]
fn a_bare_json_array_or_scalar_is_not_treated_as_json() {
// Not an object, so falls through to logfmt/none like `eval_json_comparison` would treat it.
assert_eq!(detect_format("[1,2,3]"), "none");
assert_eq!(detect_format("42"), "none");
}
#[test]
fn rejects_base64_padding_fragment_as_a_pair() {
// The exact false-positive shape from ticket 050's fixture: `key()` greedily consumes the
// tail of the base64 blob as if it were a key, landing right before the `==` padding.
let pairs = extract_pairs(
"qname: <mask.hash: 'rKTEtG7JmnThhoKPbZiVrA=='>, qtype: A, client pid: 12260 (firefox)",
);
assert!(
pairs.iter().all(|(k, _, _)| *k != "rKTEtG7JmnThhoKPbZiVrA"),
"base64 padding fragment must not be extracted as a pair: {pairs:?}"
);
}
const LOG_STREAM_FIXTURE: &str = include_str!("fixtures/log_stream_sample.txt");
#[test]
fn detects_log_stream_shaped_lines() {
let data_line = LOG_STREAM_FIXTURE.lines().nth(1).unwrap();
assert_eq!(detect_format(data_line), "log_stream");
assert!(is_log_stream_line(data_line));
}
#[test]
fn header_row_is_not_detected_as_log_stream() {
let header = LOG_STREAM_FIXTURE.lines().next().unwrap();
assert!(!is_log_stream_line(header));
}
fn column_names(header: &LogStreamHeader) -> Vec<&str> {
header.columns.iter().map(|c| c.name.as_str()).collect()
}
#[test]
fn detects_log_stream_header_columns_generically() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = detect_log_stream_header(header_line).expect("fixture header must be detected");
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type", "Activity", "PID", "TTL"]);
assert!(!header.tab_delimited);
}
#[test]
fn a_log_stream_data_line_is_never_detected_as_its_own_header() {
let data_line = LOG_STREAM_FIXTURE.lines().nth(1).unwrap();
assert!(detect_log_stream_header(data_line).is_none());
}
#[test]
fn plain_prose_is_not_detected_as_a_log_stream_header() {
assert!(detect_log_stream_header("just some plain prose here").is_none());
assert!(detect_log_stream_header("ts=1788648105773 msg=\"entity_moved\"").is_none());
}
#[test]
fn splits_log_stream_row_into_named_columns_plus_trailing_message() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = detect_log_stream_header(header_line).unwrap();
let data_line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let row = split_log_stream_row(data_line, &header);
assert_eq!(row.len(), header.columns.len() + 1);
assert_eq!(row[0], "2026-09-06 01:48:54.518117+0100");
assert_eq!(row[1], "0x4dfed2");
assert_eq!(row[2], "Default");
assert_eq!(row[3], "0x0");
assert_eq!(row[4], "606");
assert_eq!(row[5], "0");
// The message tail is never chopped up by its own internal single/double spacing.
assert_eq!(row[6], "WindowServer: (HSTouchHIDService) something happened");
}
#[test]
fn a_single_tab_separates_columns_exactly_like_a_run_of_spaces() {
let header = detect_log_stream_header("Timestamp\tThread\tType").unwrap();
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type"]);
assert!(header.tab_delimited);
// Two tabs in a row still qualify as a single separator (like a run of 2+ spaces), so this
// still cleanly separates Timestamp/Thread/Type despite the doubled tab before "Default".
let spaced = split_log_stream_row("2026-09-06 01:48:54\t0x4dfed2\t\tDefault trailing message", &header);
let tabbed = split_log_stream_row("2026-09-06 01:48:54\t0x4dfed2\tDefault trailing message", &header);
assert_eq!(spaced[0], "2026-09-06 01:48:54");
assert_eq!(spaced[1], "0x4dfed2");
assert_eq!(spaced[2], "Default trailing message");
assert_eq!(tabbed[0], "2026-09-06 01:48:54");
assert_eq!(tabbed[1], "0x4dfed2");
assert_eq!(tabbed[2], "Default trailing message");
}
#[test]
fn column_and_message_patterns_capture_the_same_fields_split_log_stream_row_finds() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = detect_log_stream_header(header_line).unwrap();
let line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let type_re = Regex::new(&log_stream_column_pattern(&header, 2, "Type")).unwrap();
assert_eq!(type_re.captures(line).unwrap().name("Type").unwrap().as_str(), "Default");
let pid_re = Regex::new(&log_stream_column_pattern(&header, 4, "PID")).unwrap();
assert_eq!(pid_re.captures(line).unwrap().name("PID").unwrap().as_str(), "606");
let message_re = Regex::new(&log_stream_message_pattern(&header, "message")).unwrap();
assert_eq!(
message_re.captures(line).unwrap().name("message").unwrap().as_str(),
"WindowServer: (HSTouchHIDService) something happened"
);
// Also exercise the Timestamp column specifically -- its own value contains one internal
// space, so a naive content-only pattern would previously over-consume into the next column.
let timestamp_re = Regex::new(&log_stream_column_pattern(&header, 0, "Timestamp")).unwrap();
assert_eq!(
timestamp_re.captures(line).unwrap().name("Timestamp").unwrap().as_str(),
"2026-09-06 01:48:54.518117+0100"
);
}
#[test]
fn a_wifi_stat_dump_line_is_still_recognized_as_log_stream_despite_embedded_key_value_pairs() {
let line = LOG_STREAM_FIXTURE
.lines()
.find(|line| line.contains("txRTSFrm="))
.expect("fixture must contain the WiFi stat dump line");
assert_eq!(detect_format(line), "log_stream");
// The embedded key=value pairs are still there for auto_declare_params to see.
assert!(!extract_pairs(line).is_empty());
}
const GO_LOG_FIXTURE: &str = include_str!("fixtures/go_log_sample.txt");
const PY_LOG_FIXTURE: &str = include_str!("fixtures/py_log_sample.txt");
#[test]
fn detects_go_log_shaped_lines() {
for line in GO_LOG_FIXTURE.lines() {
assert!(is_go_log_line(line), "expected go-log line to match: {line}");
assert_eq!(detect_format(line), "go-log");
}
}
#[test]
fn detects_py_log_shaped_lines() {
for line in PY_LOG_FIXTURE.lines() {
assert!(is_py_log_line(line), "expected py-log line to match: {line}");
assert_eq!(detect_format(line), "py-log");
}
}
#[test]
fn a_generic_json_object_is_not_detected_as_go_log() {
let line = r#"{"level":"error","code":500}"#;
assert!(!is_go_log_line(line));
assert_eq!(detect_format(line), "json");
}
#[test]
fn a_generic_logfmt_line_is_not_detected_as_py_log() {
let line = r#"ts=1788648105773 msg="entity_moved" x1=-4"#;
assert!(!is_py_log_line(line));
assert_eq!(detect_format(line), "logfmt");
}
#[test]
fn go_log_prefix_extracts_time_level_and_msg() {
let (rest, (time, level, msg)) =
go_log_prefix(r#"{"time":"2026-09-06T10:35:34.499921+01:00","level":"WARN","msg":"eventC","labelC":1}"#)
.unwrap();
assert_eq!(time, "2026-09-06T10:35:34.499921+01:00");
assert_eq!(level, "WARN");
assert_eq!(msg, "eventC");
assert_eq!(rest, r#","labelC":1}"#);
}
#[test]
fn py_log_prefix_extracts_date_time_level_and_logger() {
let (rest, (date, time, level, logger)) =
py_log_prefix("2026-09-06 10:35:55,217 INFO delete-py-log event=eventC labelC=1").unwrap();
assert_eq!(date, "2026-09-06");
assert_eq!(time, "10:35:55,217");
assert_eq!(level, "INFO");
assert_eq!(logger, "delete-py-log");
assert_eq!(rest, "event=eventC labelC=1");
}
}
src/main.rs
+4 -240
View File
@@ -1,246 +1,10 @@
mod alerts;
mod api;
mod data_value;
mod dto;
mod cli;
mod config;
mod consts;
mod engine;
mod filter_datetime;
mod filter_parser;
mod index_key;
mod ingest;
mod logfmt;
mod mcp_server;
mod metric_primitive;
mod profile;
mod promql;
mod rate;
mod sessions;
mod settings;
mod timestamp;
use std::path::PathBuf;
use std::sync::{Arc, LazyLock, Mutex};
use clap::{Parser, Subcommand};
use tokio::net::TcpListener;
use tokio::sync::mpsc;
use settings::Settings;
/// The resolved default settings path, rendered into `--help` so `--settings-path`'s help text
/// names a real, concrete location instead of only describing where to look for one.
static SETTINGS_PATH_HELP: LazyLock<String> = LazyLock::new(|| {
format!(
"Path to the settings file. Defaults to {}.",
settings::default_settings_path()
.map(|path| path.display().to_string())
.unwrap_or_else(|| "a per-OS config location".to_string())
)
});
/// Observability dashboard from terminal output.
#[derive(Parser, Debug)]
struct Args {
#[command(subcommand)]
command: Option<Command>,
/// Port to serve the API and webapp on. Defaults to 0, letting the OS assign
/// an ephemeral free port.
#[arg(long, default_value_t = 0)]
port: u16,
#[arg(long, help = SETTINGS_PATH_HELP.as_str())]
settings_path: Option<PathBuf>,
/// File to tail for log lines, instead of reading from stdin. Behaves like `tail -f`.
#[arg(long)]
file: Option<PathBuf>,
/// URL of a Prometheus/OpenMetrics-compatible endpoint to poll for metrics, in addition to log ingestion.
#[arg(long)]
metrics_url: Option<String>,
/// Name to register this session under, so `ottyl mcp --session <name>` or `ottyl sessions`
/// can find it later. Defaults to a random human-readable name (e.g. "comic-gazelle").
/// Starting a second instance under the same name as one still running overwrites its
/// registration (the older process keeps running, just without a discoverable session entry).
#[arg(long)]
name: Option<String>,
}
#[derive(Subcommand, Debug)]
enum Command {
/// Start an MCP (Model Context Protocol) server over stdio, proxying tool calls to a running
/// ottyl instance's HTTP API. Register the printed command with an MCP client (e.g. Claude
/// Code's `.mcp.json`).
Mcp {
/// Name of a running ottyl session to connect to (see `ottyl sessions`).
#[arg(long)]
session: Option<String>,
/// Port of a running ottyl instance to connect to, instead of resolving `--session` by name.
#[arg(long)]
port: Option<u16>,
},
/// List known ottyl sessions and the ports they're serving on.
Sessions,
}
#[tokio::main]
async fn main() {
let args = Args::parse();
let settings_path = args.settings_path.clone().unwrap_or_else(|| {
settings::default_settings_path()
.expect("could not determine a default settings path; pass --settings-path")
});
match args.command {
Some(Command::Mcp { session, port }) => {
run_mcp(&settings_path, session, port).await;
return;
}
Some(Command::Sessions) => {
print_sessions(&settings_path);
return;
}
None => {}
}
let settings = Arc::new(Mutex::new(Settings::load(&settings_path)));
let (engine_tx, engine_rx) = mpsc::channel(1024);
tokio::spawn(engine::run(engine_rx, settings.clone()));
{
let max_logs = settings.lock().unwrap().max_logs;
let _ = engine_tx.send(engine::EngineMessage::SetMaxLogs(max_logs)).await;
}
// Reload the active profile's declarations before ingest starts: there's no way to
// reprocess early lines later, so declaring after ingest starts could miss lines that raced ahead.
{
let (active_profile_name, active_profile) = {
let mut settings_guard = settings.lock().unwrap();
settings_guard
.ensure_default_profile(&settings_path)
.expect("failed to ensure the Default profile exists");
let active_profile_name = settings_guard.active_profile.clone();
let active_profile = active_profile_name
.as_deref()
.and_then(|name| settings_guard.profile(name).cloned());
(active_profile_name, active_profile)
};
match (active_profile_name, active_profile) {
(Some(_), Some(active_profile)) => {
profile::load_into_engine(&engine_tx, &active_profile).await;
}
(Some(profile_name), None) => {
eprintln!(
"warning: active profile '{profile_name}' not found in {}; skipping",
settings_path.display()
);
}
(None, _) => {}
}
}
match args.file.clone() {
Some(path) => {
tokio::spawn(ingest::run_file(engine_tx.clone(), path));
}
None => {
tokio::spawn(ingest::run(engine_tx.clone()));
}
}
if let Some(metrics_url) = args.metrics_url.clone() {
tokio::spawn(ingest::run_metrics_poll(engine_tx.clone(), metrics_url));
}
let app = api::router(engine_tx, settings, settings_path.clone());
let listener = TcpListener::bind(("0.0.0.0", args.port))
.await
.expect("failed to bind listener");
let bound_port = listener
.local_addr()
.expect("failed to read back bound address")
.port();
let source = match &args.file {
Some(path) => format!("tailing {}", path.display()),
None => "reading logs from stdin".to_string(),
};
let session_name = args.name.clone().unwrap_or_else(sessions::generate_name);
match sessions::register_session(&settings_path, &session_name, bound_port) {
Ok(_) => eprintln!(
"ottyl serving on http://localhost:{} ({}), session name: {}",
bound_port, source, session_name
),
Err(err) => eprintln!(
"ottyl serving on http://localhost:{} ({}); warning: failed to register session '{}': {}",
bound_port, source, session_name, err
),
}
axum::serve(listener, app)
.await
.expect("server error");
}
/// Resolves `--session`/`--port` into a target base URL and runs the MCP server over stdio until
/// the client disconnects.
async fn run_mcp(settings_path: &std::path::Path, session: Option<String>, port: Option<u16>) {
use rmcp::ServiceExt;
let port = match (port, session) {
(Some(port), _) => port,
(None, Some(name)) => match sessions::resolve_session(settings_path, &name) {
Some(info) => info.port,
None => {
eprintln!("error: no running ottyl session named '{name}'");
std::process::exit(1);
}
},
(None, None) => {
eprintln!("error: `ottyl mcp` requires --session <name> or --port <port>");
std::process::exit(1);
}
};
let base_url = format!("http://localhost:{port}");
let service = mcp_server::OttylMcp::new(base_url)
.serve(rmcp::transport::stdio())
.await
.expect("failed to start MCP server");
service.waiting().await.expect("MCP server error");
}
fn print_sessions(settings_path: &std::path::Path) {
let dir = sessions::sessions_dir(settings_path);
let sessions = sessions::list_sessions(&dir);
if sessions.is_empty() {
println!("no known ottyl sessions");
return;
}
for session in sessions {
println!("{}\tport={}\tpid={}", session.name, session.port, session.pid);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn settings_path_defaults_to_none() {
let args = Args::parse_from(["ottyl"]);
assert_eq!(args.settings_path, None);
}
#[test]
fn settings_path_flag_overrides_default() {
let args = Args::parse_from(["ottyl", "--settings-path", "/tmp/custom-settings.json"]);
assert_eq!(
args.settings_path,
Some(PathBuf::from("/tmp/custom-settings.json"))
);
}
cli::run().await;
}
tickets/070-cli-functionality-needs-fleshing-out.md
+43
View File
@@ -0,0 +1,43 @@
# 070 — CLI functionality needs to be greater — scope this out
## Summary
Wishlist feedback: "CLI functionality needs to be greater - todo
fleshout." This is a scoping ticket, not an implementation one — the
request as given doesn't say which CLI surface is thin (flags to
`ottyl` itself vs. profile/dashboard management vs. something
scriptable/headless) or what "greater" should mean in practice.
Current CLI surface, for reference: `main.rs`'s arg parsing wires
stdin/`--file` tailing, `--metrics-url` polling, and profile
selection/loading (see `settings.rs`/`profile.rs`) into engine startup.
Anything beyond piping a command and viewing the dashboard in a browser
(inspecting state, managing profiles, exporting data, scripting against
a running instance) has no CLI path today — it's all through the web
UI's API calls.
## Scope
- Talk to Hugh about what's actually missing before writing an
implementation ticket. Candidate directions worth raising:
- A way to inspect current state (params/metrics/alert rules, log
counts) from the terminal without opening the browser.
- Profile management from the CLI (list/create/switch profiles)
rather than only through the web UI's settings.
- A non-interactive/headless mode for CI-type use (e.g. run for N
seconds against piped input, dump a summary, exit) — distinct from
today's "serve until the piped process exits" model.
- Flags to declare params/metrics/dashboards at launch without
hand-editing the profile file first.
- Once scoped, split into one or more concrete tickets with specific
acceptance criteria — this ticket should be closed by replacement,
not by a partial implementation guessing at the ask.
## Acceptance criteria
- N/A — this ticket's output is a conversation and a follow-up
ticket (or tickets), not code.
## Depends on
None.
webapp/src/App.tsx
+45 -16
View File
@@ -69,6 +69,23 @@ function saveLogViewerHeight(px: number) {
}
}
// Ticket 074: a sort helper index's one-time full-scan build (`build_sort_helper_index`) runs
// inline in the engine actor, so a `sort_by` poll can occasionally take much longer than the usual
// sub-poll-interval round trip — without this, that stall looks exactly like "sorting is broken"
// rather than "still indexing". Only flips `setBuilding(true)` if the fetch is still in flight after
// `thresholdMs`, so an ordinary fast poll (helper index already built/incrementally maintained)
// never flashes the indicator.
const SORT_BUILDING_INDICATOR_THRESHOLD_MS = 400
function trackSlowSortedFetch<T>(sortBy: string | null, setBuilding: (building: boolean) => void, promise: Promise<T>): Promise<T> {
if (!sortBy) return promise
const timer = setTimeout(() => setBuilding(true), SORT_BUILDING_INDICATOR_THRESHOLD_MS)
return promise.finally(() => {
clearTimeout(timer)
setBuilding(false)
})
}
/// Thin drag handle between the log viewer and the sections below it. Reports each pointer-move
/// delta via `onResize` (letting the caller clamp/apply it) and fires `onResizeEnd` on release so
/// the final size can be persisted once, not on every pixel of movement.
@@ -286,6 +303,7 @@ function StandaloneLog() {
const [filter, setFilter] = useState('')
const [filterError, setFilterError] = useState<string | null>(null)
const [sortSettings, setSortSettings] = useState<LogSortSettings>(loadLogSortSettings)
const [sortIndexBuilding, setSortIndexBuilding] = useState(false)
function handleSortSettingsChange(next: LogSortSettings) {
setSortSettings(next)
@@ -341,14 +359,18 @@ function StandaloneLog() {
bucketMs: effectiveRange ? computeBucketMs(effectiveRange.endMs - effectiveRange.startMs) : undefined,
filter: filter || undefined,
}),
fetchLogs({
startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs,
limit: 1000,
filter: filter || undefined,
sortBy: sortSettings.sortBy ?? undefined,
sortMissingFallback: sortSettings.sortBy ? sortSettings.sortMissingFallback : undefined,
}),
trackSlowSortedFetch(
sortSettings.sortBy,
setSortIndexBuilding,
fetchLogs({
startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs,
limit: 1000,
filter: filter || undefined,
sortBy: sortSettings.sortBy ?? undefined,
sortMissingFallback: sortSettings.sortBy ? sortSettings.sortMissingFallback : undefined,
}),
),
// Ticket 058: params can get auto-declared from freshly ingested lines mid-run, not just from a user's own edit.
fetchParamDeclarations(),
])
@@ -393,6 +415,7 @@ function StandaloneLog() {
paramNames={params.map((p) => p.name)}
sortSettings={sortSettings}
onSortSettingsChange={handleSortSettingsChange}
sortIndexBuilding={sortIndexBuilding}
/>
<LogPreview logs={logs} params={params} />
</div>
@@ -423,6 +446,7 @@ function FullApp() {
const [logViewerCollapsed, setLogViewerCollapsed] = useSectionCollapsed('log-viewer')
const [logViewerHeight, setLogViewerHeight] = useState(() => loadLogViewerHeight())
const [sortSettings, setSortSettings] = useState<LogSortSettings>(loadLogSortSettings)
const [sortIndexBuilding, setSortIndexBuilding] = useState(false)
function handleSortSettingsChange(next: LogSortSettings) {
setSortSettings(next)
@@ -497,14 +521,18 @@ function FullApp() {
bucketMs: effectiveRange ? computeBucketMs(effectiveRange.endMs - effectiveRange.startMs) : undefined,
filter: filter || undefined,
}),
fetchLogs({
startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs,
limit: 1000,
filter: filter || undefined,
sortBy: sortSettings.sortBy ?? undefined,
sortMissingFallback: sortSettings.sortBy ? sortSettings.sortMissingFallback : undefined,
}),
trackSlowSortedFetch(
sortSettings.sortBy,
setSortIndexBuilding,
fetchLogs({
startMs: effectiveRange?.startMs,
endMs: effectiveRange?.endMs,
limit: 1000,
filter: filter || undefined,
sortBy: sortSettings.sortBy ?? undefined,
sortMissingFallback: sortSettings.sortBy ? sortSettings.sortMissingFallback : undefined,
}),
),
// Ticket 058: params can get auto-declared from freshly ingested lines mid-run, not just from a user's own edit.
fetchParamDeclarations(),
])
@@ -566,6 +594,7 @@ function FullApp() {
paramNames={params.map((p) => p.name)}
sortSettings={sortSettings}
onSortSettingsChange={handleSortSettingsChange}
sortIndexBuilding={sortIndexBuilding}
/>
<LogPreview logs={logs} params={params} />
</div>
webapp/src/components/Docs.tsx
+99 -66
View File
@@ -78,75 +78,108 @@ const SECTIONS: DocSection[] = [
id: 'filtering',
title: 'Filtering',
body: (
<div className="flex flex-col gap-2">
<div className="flex flex-col gap-3">
<p>
The filter bar combines regex search over raw log text with comparisons against declared metrics, joined
by <Code>and</Code>/<Code>or</Code>. Examples:
by <Code>and</Code>/<Code>or</Code>. The server evaluates the filter string; a parse error shows as inline
text under the filter bar.
</p>
<ul className="list-disc pl-5">
<li>
<Code>{'regex("error")'}</Code> — lines containing "error".
</li>
<li>
<Code>{'regex("error") and ottyl_seq<5'}</Code> — lines containing "error" with sequence under 5.
</li>
<li>
<Code>{'requests[duration]>100'}</Code> — the <Code>requests</Code> metric's captured{' '}
<Code>duration</Code> param over 100.
</li>
<li>
<Code>{'ottyl_ts>-5m'}</Code> — lines from the last 5 minutes.
</li>
<li>
<Code>{'ottyl_fmt==json'}</Code> — lines auto-detected as JSON.
</li>
</ul>
<p>
<Code>ottyl_seq</Code>, <Code>ottyl_ts</Code>, and <Code>ottyl_fmt</Code> are reserved metric names present
on every log line, so they're always filterable without declaring anything. <Code>ottyl_fmt</Code> holds the
auto-detected format of the line's text — <Code>log_stream</Code> (macOS <Code>log stream</Code> output),{' '}
<Code>json</Code>, <Code>logfmt</Code>, or <Code>none</Code>. A <Code>log_stream</Code> line also gets{' '}
<Code>log_type</Code>, <Code>process</Code>, and <Code>pid</Code> params derived automatically from its
fixed-column header. All three reserved names also appear (flagged <Code>reserved</Code>) in{' '}
<Code>GET /api/params</Code> for discoverability, though there's no UI to edit or remove them — see also{' '}
<em>Dashboards</em> below for using <Code>ottyl_seq</Code>/<Code>ottyl_ts</Code> in a panel query.
</p>
<p className="font-semibold text-text">Date/time formats</p>
<p>Timestamp comparisons accept three forms:</p>
<ul className="list-disc pl-5">
<li>
relative offsets from now — <Code>5h</Code> (5 hours from now), <Code>-3m</Code> (3 minutes ago)
</li>
<li>
absolute clock times — <Code>12:00</Code>, <Code>09:30:00</Code>
</li>
<li>
full dates — <Code>2026-01-15</Code>, <Code>2026-01-15T09:30:00Z</Code>
</li>
</ul>
<p>
The filter string is evaluated on the server. A parse error shows as inline text under the filter bar.
</p>
<p className="font-semibold text-text">Worked examples</p>
<ul className="list-disc pl-5">
<li>
Only lines mentioning "timeout" in the last 15 minutes:{' '}
<Code>{'regex("timeout") and ottyl_ts>-15m'}</Code>
</li>
<li>
Requests slower than 200ms, once a <Code>requests</Code> metric has a <Code>value</Code>-role{' '}
<Code>duration</Code> param declared: <Code>{'requests[duration]>200'}</Code>
</li>
<li>
5xx responses on one path, given a <Code>requests</Code> metric with a <Code>path</Code> label param —
note that <Code>[param]</Code> comparisons only work against a <em>declared metric's</em> labels, not a
bare param declared on its own, and values are unquoted (no surrounding <Code>"..."</Code>):{' '}
<Code>{'requests[path]==/checkout and regex("5[0-9]{2}")'}</Code>
</li>
<li>
Everything before noon on a specific date: <Code>{'ottyl_ts<2024-06-15T12:00:00Z'}</Code>
</li>
</ul>
<div>
<p className="font-semibold text-text">Regex match</p>
<ul className="list-disc pl-5">
<li>
<Code>{'regex("error")'}</Code> — lines containing "error". See the <Code>@</Code> icon next to the
filter bar for the exact regex syntax supported (Rust's <Code>regex</Code> crate — no lookaround or
backreferences).
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Metric/param comparisons</p>
<ul className="list-disc pl-5">
<li>
<Code>{'requests[duration]>100'}</Code> — the <Code>requests</Code> metric's captured{' '}
<Code>duration</Code> param over 100. <Code>[param]</Code> comparisons only work against a{' '}
<em>declared metric's</em> labels, not a bare param declared on its own, and values are unquoted (no
surrounding <Code>"..."</Code>).
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Reserved names (always filterable)</p>
<p>
<Code>ottyl_seq</Code>, <Code>ottyl_ts</Code>, and <Code>ottyl_fmt</Code> are present on every log line
without declaring anything, and also appear (flagged <Code>reserved</Code>) in{' '}
<Code>GET /api/params</Code> for discoverability, though there's no UI to edit or remove them.
</p>
<ul className="list-disc pl-5">
<li>
<Code>{'ottyl_seq<5'}</Code> — ingest sequence number under 5.
</li>
<li>
<Code>{'ottyl_ts>-5m'}</Code> — lines from the last 5 minutes (see Date/time formats below).
</li>
<li>
<Code>{'ottyl_fmt==json'}</Code> — the line's auto-detected text format:{' '}
<Code>log_stream</Code> (macOS <Code>log stream</Code> output — its columns are also auto-declared as
their own named params, e.g. <Code>PID</Code>/<Code>Thread</Code>, from that line's own header), <Code>json</Code>,{' '}
<Code>logfmt</Code>, or <Code>none</Code>.
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Date/time formats</p>
<p>Timestamp comparisons (e.g. against <Code>ottyl_ts</Code>) accept three forms:</p>
<ul className="list-disc pl-5">
<li>
relative offsets from now — <Code>5h</Code> (5 hours from now), <Code>-3m</Code> (3 minutes ago)
</li>
<li>
absolute clock times — <Code>12:00</Code>, <Code>09:30:00</Code>
</li>
<li>
full dates — <Code>2026-01-15</Code>, <Code>2026-01-15T09:30:00Z</Code>
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Combining clauses</p>
<ul className="list-disc pl-5">
<li>
<Code>{'regex("error") and ottyl_seq<5'}</Code> — lines containing "error" with sequence under 5.
</li>
<li>
<Code>{'requests[path]==/checkout and regex("5[0-9]{2}")'}</Code> — 5xx responses on one path, given a{' '}
<Code>requests</Code> metric with a <Code>path</Code> label param.
</li>
</ul>
</div>
<div>
<p className="font-semibold text-text">Worked examples</p>
<ul className="list-disc pl-5">
<li>
Only lines mentioning "timeout" in the last 15 minutes:{' '}
<Code>{'regex("timeout") and ottyl_ts>-15m'}</Code>
</li>
<li>
Requests slower than 200ms, once a <Code>requests</Code> metric has a <Code>value</Code>-role{' '}
<Code>duration</Code> param declared: <Code>{'requests[duration]>200'}</Code>
</li>
<li>
Everything before noon on a specific date: <Code>{'ottyl_ts<2024-06-15T12:00:00Z'}</Code>
</li>
</ul>
<p className="mt-1 text-text-subtle">
See also <em>Dashboards</em> below for using <Code>ottyl_seq</Code>/<Code>ottyl_ts</Code> in a panel
query.
</p>
</div>
</div>
),
},
webapp/src/components/FilterBar.tsx
+5 -4
View File
@@ -41,6 +41,11 @@ export function FilterBar({
<span className="flex items-center gap-1">
<HelpIcon sectionId="filtering" label="Help: filter syntax" />
<span className="text-xs text-text-subtle">Filter syntax</span>
<span className="mx-0.5 h-4 w-px bg-border-soft" aria-hidden="true" />
<RegexHelpIcon label="Help: regex(...) syntax" />
<span className="text-xs text-text-faint" title="Cheat sheet for regex(...) clauses specifically">
regex(...)
</span>
</span>
{collapseToggle}
<input
@@ -49,10 +54,6 @@ export function FilterBar({
value={draft}
onChange={(e) => setDraft(e.target.value)}
/>
<span className="flex items-center gap-1" title="Cheat sheet for regex(...) clauses specifically">
<RegexHelpIcon label="Help: regex(...) syntax" />
<span className="text-xs text-text-faint">regex(...)</span>
</span>
<button
type="submit"
className="rounded border border-border-soft px-3 py-1 text-sm text-text-secondary hover:bg-surface-2"
webapp/src/components/LogTimeline.tsx
+9
View File
@@ -141,6 +141,7 @@ export function LogTimeline({
paramNames,
sortSettings,
onSortSettingsChange,
sortIndexBuilding,
}: {
buckets: TimelineBucket[]
range: TimeRange | null
@@ -151,6 +152,9 @@ export function LogTimeline({
paramNames?: string[]
sortSettings?: LogSortSettings
onSortSettingsChange?: (next: LogSortSettings) => void
/** Ticket 074: set by the caller when a `sort_by` fetch has been in flight long enough that it's
* likely the engine is running `build_sort_helper_index`'s one-time full scan, not just stalled. */
sortIndexBuilding?: boolean
}) {
const containerRef = useRef<HTMLDivElement>(null)
const [dragStartX, setDragStartX] = useState<number | null>(null)
@@ -224,6 +228,11 @@ export function LogTimeline({
{sortSettings && onSortSettingsChange && (
<SortPicker paramNames={paramNames ?? []} settings={sortSettings} onChange={onSortSettingsChange} />
)}
{sortIndexBuilding && (
<span className="text-text-muted" title="First sort by this param scans every ingested line once; later polls reuse the built index.">
Building sort index…
</span>
)}
<div className="flex items-center gap-1.5">
<div className="flex items-center gap-1">
{TIME_RANGE_PRESETS.map((preset) => (
webapp/src/components/NavMenu.tsx
+13
View File
@@ -41,6 +41,16 @@ function schemeLabel(scheme: ColorScheme): string {
return scheme.replace(/_/g, ' ')
}
/** Ticket 073: plain `Xms` under a second (this stat is normally near-zero, so that's the common case), a humantime-style `Xs`/`Xm` above it — mirrors the backend's own humantime-duration inputs (see `Docs.tsx`'s window-parsing note) without pulling in a full duration-formatting library for one stat. */
function formatIngestLag(ms: number): string {
if (ms < 1000) return `${ms}ms`
const totalSeconds = ms / 1000
if (totalSeconds < 60) return `${totalSeconds % 1 === 0 ? totalSeconds : totalSeconds.toFixed(1)}s`
const minutes = Math.floor(totalSeconds / 60)
const seconds = Math.round(totalSeconds % 60)
return seconds === 0 ? `${minutes}m` : `${minutes}m${seconds}s`
}
export function NavMenu({ stats }: { stats: Stats | null }) {
const [openPanel, setOpenPanel] = useState<PanelName>(null)
const panel = useSettingsPanel()
@@ -102,6 +112,9 @@ export function NavMenu({ stats }: { stats: Stats | null }) {
seq {stats.first_sequence ?? 0}–{stats.last_sequence ?? 0}
</div>
<div>{stats.approx_bytes_total.toLocaleString()} bytes</div>
<div title="Highest ingestion-to-engine-processing lag seen in the last 5s">
lag {formatIngestLag(stats.ingest_lag_high_watermark_ms)}
</div>
</>
) : (
<div>connecting…</div>
webapp/src/lib/api.ts
+2
View File
@@ -49,6 +49,8 @@ export interface Stats {
last_timestamp_ms: number | null
/** Ticket 011: running approximate byte size of all currently-ingested logs. */
approx_bytes_total: number
/** Ticket 073: highest ingestion-to-engine-processing lag (ms) seen within the trailing 5s. */
ingest_lag_high_watermark_ms: number
}
export interface MetricDeclaration {