phughk/ottyl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Adopt rustfmt and clean up clippy warnings

Browse Source 
Formats the whole backend to rustfmt defaults and fixes all 23
outstanding clippy warnings (collapsible ifs, clone-to-slice-ref,
type-complexity, single-arm match, unnecessary is_none check), then
wires cargo fmt --check and -D warnings into make lint so both stay
enforced going forward.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TeyhRfMYjMditek6geDRoQ
This commit is contained in:
Hugh Kaznowski
2026-09-07 12:21:38 +01:00
co-authored by Claude Sonnet 5
parent 2cc8ac8138
commit c3108677ce
28 changed files with 4721 additions and 2851 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+2 -1
View File
@@ -23,7 +23,8 @@ test-fe:
lint: lint-be lint-fe
lint-be:
cargo clippy --all-targets
cargo fmt -- --check
cargo clippy --all-targets -- -D warnings
lint-fe:
cd webapp && npm run lint
src/api/dto.rs
+2 -2
View File
@@ -1,8 +1,8 @@
use crate::config::Settings;
use crate::config::profile::{Panel, Profile};
use crate::engine::alerts::AlertCondition;
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::metric_primitive::MetricPrimitive;
use crate::config::Settings;
use crate::config::profile::{Panel, Profile};
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
src/api/mcp_server.rs
+142 -44
View File
@@ -381,7 +381,9 @@ impl OttylMcp {
// ---- logs / timeline / stats ----
#[tool(description = "Count or read log lines matching a FilterBar query against the running ottyl instance.")]
#[tool(
description = "Count or read log lines matching a FilterBar query against the running ottyl instance."
)]
async fn query_logs(
&self,
Parameters(params): Parameters<QueryLogsParams>,
@@ -414,7 +416,9 @@ impl OttylMcp {
}
}
#[tool(description = "Read the log-count timeline (bucketed histogram) matching an optional FilterBar query — the same data backing the Log Timeline chart.")]
#[tool(
description = "Read the log-count timeline (bucketed histogram) matching an optional FilterBar query — the same data backing the Log Timeline chart."
)]
async fn get_timeline(
&self,
Parameters(params): Parameters<GetTimelineParams>,
@@ -435,14 +439,18 @@ impl OttylMcp {
json_result(self.get_json("/api/timeline", &query).await?)
}
#[tool(description = "Read overall ingest stats: log count, sequence/timestamp bounds, ingest lag.")]
#[tool(
description = "Read overall ingest stats: log count, sequence/timestamp bounds, ingest lag."
)]
async fn get_stats(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/stats", &[]).await?)
}
// ---- params ----
#[tool(description = "List all currently declared params (name, regex, kind, etc.), including auto-derived ones.")]
#[tool(
description = "List all currently declared params (name, regex, kind, etc.), including auto-derived ones."
)]
async fn list_params(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/params", &[]).await?)
}
@@ -452,13 +460,22 @@ impl OttylMcp {
&self,
Parameters(params): Parameters<DeclareParamParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/params", Some(&body)).await?)
let body = serde_json::to_value(&params)
.map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(reqwest::Method::POST, "/api/params", Some(&body))
.await?,
)
}
#[tool(description = "Clear all declared params and re-derive auto params from retained log lines.")]
#[tool(
description = "Clear all declared params and re-derive auto params from retained log lines."
)]
async fn clear_params(&self) -> Result<CallToolResult, McpError> {
json_result(self.send_json(reqwest::Method::DELETE, "/api/params", None).await?)
json_result(
self.send_json(reqwest::Method::DELETE, "/api/params", None)
.await?,
)
}
// ---- metrics ----
@@ -468,13 +485,19 @@ impl OttylMcp {
json_result(self.get_json("/api/metrics", &[]).await?)
}
#[tool(description = "Declare a metric: named event counts/values bucketed by label, derived from matching log lines.")]
#[tool(
description = "Declare a metric: named event counts/values bucketed by label, derived from matching log lines."
)]
async fn declare_metric(
&self,
Parameters(params): Parameters<DeclareMetricParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/metrics", Some(&body)).await?)
let body = serde_json::to_value(&params)
.map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(reqwest::Method::POST, "/api/metrics", Some(&body))
.await?,
)
}
#[tool(description = "Update an existing metric declaration in place by name.")]
@@ -509,7 +532,9 @@ impl OttylMcp {
)
}
#[tool(description = "Read a metric's raw bucket values (labels, sequence, timestamp, value), optionally filtered by label and time range.")]
#[tool(
description = "Read a metric's raw bucket values (labels, sequence, timestamp, value), optionally filtered by label and time range."
)]
async fn get_metric_bucket_values(
&self,
Parameters(params): Parameters<GetMetricBucketValuesParams>,
@@ -531,19 +556,27 @@ impl OttylMcp {
query.push(("limit", limit.to_string()));
}
json_result(
self.get_json(&format!("/api/metrics/{}", urlencoding_path(&params.name)), &query)
.await?,
self.get_json(
&format!("/api/metrics/{}", urlencoding_path(&params.name)),
&query,
)
.await?,
)
}
// ---- promql ----
#[tool(description = "Evaluate a PromQL-style query directly and get back raw series data (labels + timestamped points) — the same evaluation dashboards use to render charts.")]
#[tool(
description = "Evaluate a PromQL-style query directly and get back raw series data (labels + timestamped points) — the same evaluation dashboards use to render charts."
)]
async fn run_promql_query(
&self,
Parameters(params): Parameters<RunPromqlQueryParams>,
) -> Result<CallToolResult, McpError> {
json_result(self.run_promql(&params.promql, params.start_ms, params.end_ms).await?)
json_result(
self.run_promql(&params.promql, params.start_ms, params.end_ms)
.await?,
)
}
// ---- dashboards / panels ----
@@ -556,7 +589,10 @@ impl OttylMcp {
let profile = self.resolve_profile(params.profile).await?;
json_result(
self.get_json(
&format!("/api/settings/profiles/{}/dashboards", urlencoding_path(&profile)),
&format!(
"/api/settings/profiles/{}/dashboards",
urlencoding_path(&profile)
),
&[],
)
.await?,
@@ -572,14 +608,19 @@ impl OttylMcp {
json_result(
self.send_json(
reqwest::Method::POST,
&format!("/api/settings/profiles/{}/dashboards", urlencoding_path(&profile)),
&format!(
"/api/settings/profiles/{}/dashboards",
urlencoding_path(&profile)
),
Some(&serde_json::json!({ "name": params.name })),
)
.await?,
)
}
#[tool(description = "Read a dashboard's current panels (definitions only — use render_panel or render_dashboard for the data a panel would chart).")]
#[tool(
description = "Read a dashboard's current panels (definitions only — use render_panel or render_dashboard for the data a panel would chart)."
)]
async fn get_dashboard(
&self,
Parameters(params): Parameters<GetDashboardParams>,
@@ -631,12 +672,18 @@ impl OttylMcp {
.get_mut("panels")
.and_then(Value::as_array_mut)
.ok_or_else(|| mcp_error("unexpected dashboard shape", "missing panels array"))?;
match panels.iter().position(|panel| panel.get("id") == new_panel.get("id")) {
match panels
.iter()
.position(|panel| panel.get("id") == new_panel.get("id"))
{
Some(index) => panels[index] = new_panel,
None => panels.push(new_panel),
}
json_result(self.save_dashboard(&profile, &params.dashboard_id, dashboard).await?)
json_result(
self.save_dashboard(&profile, &params.dashboard_id, dashboard)
.await?,
)
}
#[tool(description = "Remove a panel from a dashboard by panel_id.")]
@@ -650,8 +697,13 @@ impl OttylMcp {
.get_mut("panels")
.and_then(Value::as_array_mut)
.ok_or_else(|| mcp_error("unexpected dashboard shape", "missing panels array"))?;
panels.retain(|panel| panel.get("id").and_then(Value::as_str) != Some(params.panel_id.as_str()));
json_result(self.save_dashboard(&profile, &params.dashboard_id, dashboard).await?)
panels.retain(|panel| {
panel.get("id").and_then(Value::as_str) != Some(params.panel_id.as_str())
});
json_result(
self.save_dashboard(&profile, &params.dashboard_id, dashboard)
.await?,
)
}
async fn save_dashboard(
@@ -672,7 +724,9 @@ impl OttylMcp {
.await
}
#[tool(description = "Render one panel: resolve its promql from the dashboard definition, evaluate it, and return the chart's underlying series data (labels + timestamped points) as JSON — this is how an agent should 'see' a chart without a browser.")]
#[tool(
description = "Render one panel: resolve its promql from the dashboard definition, evaluate it, and return the chart's underlying series data (labels + timestamped points) as JSON — this is how an agent should 'see' a chart without a browser."
)]
async fn render_panel(
&self,
Parameters(params): Parameters<RenderPanelParams>,
@@ -682,20 +736,30 @@ impl OttylMcp {
let panel = dashboard
.get("panels")
.and_then(Value::as_array)
.and_then(|panels| panels.iter().find(|panel| panel.get("id").and_then(Value::as_str) == Some(params.panel_id.as_str())))
.ok_or_else(|| McpError::invalid_params("no panel with that panel_id in the dashboard", None))?;
.and_then(|panels| {
panels.iter().find(|panel| {
panel.get("id").and_then(Value::as_str) == Some(params.panel_id.as_str())
})
})
.ok_or_else(|| {
McpError::invalid_params("no panel with that panel_id in the dashboard", None)
})?;
let promql = panel
.get("promql")
.and_then(Value::as_str)
.ok_or_else(|| mcp_error("unexpected panel shape", "missing promql"))?;
let series = self.run_promql(promql, params.start_ms, params.end_ms).await?;
let series = self
.run_promql(promql, params.start_ms, params.end_ms)
.await?;
json_result(serde_json::json!({
"panel": panel,
"series": series,
}))
}
#[tool(description = "Render every panel in a dashboard: evaluate each panel's promql and return all their chart data as JSON, so an agent can reason about the whole dashboard without a browser.")]
#[tool(
description = "Render every panel in a dashboard: evaluate each panel's promql and return all their chart data as JSON, so an agent can reason about the whole dashboard without a browser."
)]
async fn render_dashboard(
&self,
Parameters(params): Parameters<RenderDashboardParams>,
@@ -709,8 +773,13 @@ impl OttylMcp {
.unwrap_or_default();
let mut rendered = Vec::with_capacity(panels.len());
for panel in panels {
let promql = panel.get("promql").and_then(Value::as_str).unwrap_or_default();
let series = self.run_promql(promql, params.start_ms, params.end_ms).await?;
let promql = panel
.get("promql")
.and_then(Value::as_str)
.unwrap_or_default();
let series = self
.run_promql(promql, params.start_ms, params.end_ms)
.await?;
rendered.push(serde_json::json!({ "panel": panel, "series": series }));
}
json_result(serde_json::json!({
@@ -721,18 +790,26 @@ impl OttylMcp {
// ---- settings / theming ----
#[tool(description = "Read the running instance's global settings (theme, active profile, max_logs, ntfy config).")]
#[tool(
description = "Read the running instance's global settings (theme, active profile, max_logs, ntfy config)."
)]
async fn get_settings(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/settings", &[]).await?)
}
#[tool(description = "Update global settings. Only fields provided are changed; pass null for an Option field to clear it.")]
#[tool(
description = "Update global settings. Only fields provided are changed; pass null for an Option field to clear it."
)]
async fn update_settings(
&self,
Parameters(params): Parameters<UpdateSettingsParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::PATCH, "/api/settings", Some(&body)).await?)
let body = serde_json::to_value(&params)
.map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(reqwest::Method::PATCH, "/api/settings", Some(&body))
.await?,
)
}
#[tool(description = "List the names of the themes embedded in this build (see /api/themes).")]
@@ -779,7 +856,10 @@ impl OttylMcp {
"enabled": params.enabled,
"condition": params.condition,
});
json_result(self.send_json(reqwest::Method::POST, "/api/settings/alerts", Some(&body)).await?)
json_result(
self.send_json(reqwest::Method::POST, "/api/settings/alerts", Some(&body))
.await?,
)
}
#[tool(description = "Update an existing alert rule by id.")]
@@ -817,21 +897,30 @@ impl OttylMcp {
)
}
#[tool(description = "Send a test ntfy notification to verify alert delivery is configured correctly.")]
#[tool(
description = "Send a test ntfy notification to verify alert delivery is configured correctly."
)]
async fn test_alert_notification(
&self,
Parameters(params): Parameters<TestAlertNotificationParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
let body = serde_json::to_value(&params)
.map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(reqwest::Method::POST, "/api/settings/alerts/test", Some(&body))
.await?,
self.send_json(
reqwest::Method::POST,
"/api/settings/alerts/test",
Some(&body),
)
.await?,
)
}
// ---- profiles ----
#[tool(description = "List all saved profiles with summary counts (params, metrics, dashboards).")]
#[tool(
description = "List all saved profiles with summary counts (params, metrics, dashboards)."
)]
async fn list_profiles(&self) -> Result<CallToolResult, McpError> {
json_result(self.get_json("/api/settings/profiles", &[]).await?)
}
@@ -841,8 +930,12 @@ impl OttylMcp {
&self,
Parameters(params): Parameters<CreateProfileParams>,
) -> Result<CallToolResult, McpError> {
let body = serde_json::to_value(&params).map_err(|err| mcp_error("failed to encode params", err))?;
json_result(self.send_json(reqwest::Method::POST, "/api/settings/profiles", Some(&body)).await?)
let body = serde_json::to_value(&params)
.map_err(|err| mcp_error("failed to encode params", err))?;
json_result(
self.send_json(reqwest::Method::POST, "/api/settings/profiles", Some(&body))
.await?,
)
}
#[tool(description = "Delete a profile by name.")]
@@ -860,7 +953,9 @@ impl OttylMcp {
)
}
#[tool(description = "Activate a profile: swaps in its param/metric declarations and dashboards as the running instance's live state.")]
#[tool(
description = "Activate a profile: swaps in its param/metric declarations and dashboards as the running instance's live state."
)]
async fn activate_profile(
&self,
Parameters(params): Parameters<ProfileNameParams>,
@@ -868,7 +963,10 @@ impl OttylMcp {
json_result(
self.send_json(
reqwest::Method::POST,
&format!("/api/settings/profiles/{}/activate", urlencoding_path(&params.name)),
&format!(
"/api/settings/profiles/{}/activate",
urlencoding_path(&params.name)
),
None,
)
.await?,
src/api/mod.rs
+342 -82
View File
@@ -7,23 +7,23 @@ use std::sync::Arc;
use crate::api::dto::{
BucketValuesParams, CreateAlertRuleRequest, CreateDashboardRequest, CreateProfileRequest,
ErrorDto, MetricDeclarationDto,
ParamDeclarationDto, LogsParams, ProfileSummaryDto, PromqlParams, SaveDashboardRequest,
SettingsDto, TestAlertNotificationRequest, TimelineParams, UpdateAlertRuleRequest, UpdateSettingsRequest,
ErrorDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto, ProfileSummaryDto,
PromqlParams, SaveDashboardRequest, SettingsDto, TestAlertNotificationRequest, TimelineParams,
UpdateAlertRuleRequest, UpdateSettingsRequest,
};
use crate::config::SettingsHandle;
use crate::config::profile::{Dashboard, ParamDeclarationRecord, Profile, ProfileError};
use crate::engine::alerts::{AlertError, deliver_ntfy};
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::{
DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage, EngineQuery,
EngineResponse,
};
use crate::engine::alerts::{deliver_ntfy, AlertError};
use crate::config::profile::{Dashboard, ParamDeclarationRecord, Profile, ProfileError};
use crate::config::SettingsHandle;
use axum::Router;
use axum::extract::{Path as AxumPath, Query, State};
use axum::http::{header, StatusCode};
use axum::http::{StatusCode, header};
use axum::response::{IntoResponse, Json, Response};
use axum::routing::{delete, get, post, put};
use axum::Router;
use fs_embed::Dir as EmbedDir;
use tokio::sync::oneshot;
@@ -49,7 +49,9 @@ impl ApiState {
let Some(profile_name) = settings.active_profile.clone() else {
return;
};
if let Err(err) = settings.record_param_declaration(&self.settings_path, &profile_name, request) {
if let Err(err) =
settings.record_param_declaration(&self.settings_path, &profile_name, request)
{
eprintln!(
"warning: failed to persist param declaration '{}' into profile '{profile_name}': {err}",
request.name
@@ -79,7 +81,10 @@ impl ApiState {
};
let existing_params = {
let settings = self.settings.lock().unwrap();
settings.profile(&profile_name).map(|p| p.params.clone()).unwrap_or_default()
settings
.profile(&profile_name)
.map(|p| p.params.clone())
.unwrap_or_default()
};
let promoted = promoted_param_records(&self.engine, &existing_params, request).await;
let mut settings = self.settings.lock().unwrap();
@@ -96,7 +101,11 @@ impl ApiState {
}
}
async fn persist_metric_declaration_update(&self, old_name: &str, request: &DeclareMetricRequest) {
async fn persist_metric_declaration_update(
&self,
old_name: &str,
request: &DeclareMetricRequest,
) {
let profile_name = {
let settings = self.settings.lock().unwrap();
let Some(profile_name) = settings.active_profile.clone() else {
@@ -106,7 +115,10 @@ impl ApiState {
};
let existing_params = {
let settings = self.settings.lock().unwrap();
settings.profile(&profile_name).map(|p| p.params.clone()).unwrap_or_default()
settings
.profile(&profile_name)
.map(|p| p.params.clone())
.unwrap_or_default()
};
let promoted = promoted_param_records(&self.engine, &existing_params, request).await;
let mut settings = self.settings.lock().unwrap();
@@ -129,7 +141,9 @@ impl ApiState {
let Some(profile_name) = settings.active_profile.clone() else {
return;
};
if let Err(err) = settings.remove_metric_declaration(&self.settings_path, &profile_name, name) {
if let Err(err) =
settings.remove_metric_declaration(&self.settings_path, &profile_name, name)
{
eprintln!(
"warning: failed to remove metric declaration '{name}' from profile '{profile_name}': {err}"
);
@@ -149,39 +163,58 @@ pub fn router(engine: EngineHandle, settings: SettingsHandle, settings_path: Pat
.route("/api/logs", get(get_logs))
.route("/api/timeline", get(get_timeline))
.route("/api/stats", get(get_stats))
.route("/api/params", get(list_params).post(declare_param).delete(clear_params))
.route(
"/api/params",
get(list_params).post(declare_param).delete(clear_params),
)
.route("/api/metrics", get(list_metrics).post(declare_metric))
.route(
"/api/metrics/{name}",
get(get_metric_bucket_values).put(update_metric).delete(delete_metric),
get(get_metric_bucket_values)
.put(update_metric)
.delete(delete_metric),
)
.route("/api/query", get(run_promql_query))
.route("/api/settings", get(get_settings).patch(update_settings))
.route("/api/themes", get(list_themes))
.route("/api/themes/{name}", get(get_theme_content))
.route("/api/settings/alerts", get(list_alert_rules).post(create_alert_rule))
.route(
"/api/settings/alerts",
get(list_alert_rules).post(create_alert_rule),
)
.route(
"/api/settings/alerts/{id}",
put(update_alert_rule).delete(delete_alert_rule),
)
.route("/api/settings/alerts/test", post(test_alert_notification))
.route("/api/settings/profiles", get(list_profiles).post(create_profile))
.route(
"/api/settings/profiles",
get(list_profiles).post(create_profile),
)
.route("/api/settings/profiles/{name}", delete(delete_profile))
.route("/api/settings/profiles/{name}/activate", post(activate_profile))
.route(
"/api/settings/profiles/{name}/activate",
post(activate_profile),
)
.route(
"/api/settings/profiles/{name}/dashboards",
get(list_dashboards).post(create_dashboard),
)
.route(
"/api/settings/profiles/{name}/dashboards/{dashboard_id}",
get(get_dashboard).put(save_dashboard).delete(delete_dashboard),
get(get_dashboard)
.put(save_dashboard)
.delete(delete_dashboard),
)
.with_state(state)
.fallback(serve_webapp)
}
async fn get_skill_md() -> impl IntoResponse {
([(header::CONTENT_TYPE, "text/markdown; charset=utf-8")], SKILL_MD)
(
[(header::CONTENT_TYPE, "text/markdown; charset=utf-8")],
SKILL_MD,
)
}
async fn serve_webapp(uri: axum::http::Uri) -> Response {
@@ -225,10 +258,15 @@ impl EngineHandle {
if needs_build {
let (respond_to, response) = oneshot::channel();
self.tx
.send(EngineMessage::EnsureSortHelperIndex { param_name: sort_by.to_string(), respond_to })
.send(EngineMessage::EnsureSortHelperIndex {
param_name: sort_by.to_string(),
respond_to,
})
.await
.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;
response
.await
.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;
response.await.map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;
}
}
let guard = read_handle.enter().ok_or(StatusCode::SERVICE_UNAVAILABLE)?;
@@ -363,7 +401,10 @@ async fn declare_param(
if state
.engine
.tx
.send(EngineMessage::DeclareParam { request: body, respond_to })
.send(EngineMessage::DeclareParam {
request: body,
respond_to,
})
.await
.is_err()
{
@@ -391,7 +432,13 @@ async fn declare_param(
async fn clear_params(State(state): State<ApiState>) -> impl IntoResponse {
let (respond_to, response) = oneshot::channel();
if state.engine.tx.send(EngineMessage::ClearParams { respond_to }).await.is_err() {
if state
.engine
.tx
.send(EngineMessage::ClearParams { respond_to })
.await
.is_err()
{
return StatusCode::SERVICE_UNAVAILABLE.into_response();
}
match response.await {
@@ -420,7 +467,10 @@ async fn declare_metric(
if state
.engine
.tx
.send(EngineMessage::DeclareMetric { request: body, respond_to })
.send(EngineMessage::DeclareMetric {
request: body,
respond_to,
})
.await
.is_err()
{
@@ -454,7 +504,11 @@ async fn update_metric(
if state
.engine
.tx
.send(EngineMessage::UpdateMetric { name: name.clone(), request: body, respond_to })
.send(EngineMessage::UpdateMetric {
name: name.clone(),
request: body,
respond_to,
})
.await
.is_err()
{
@@ -462,7 +516,9 @@ async fn update_metric(
}
match response.await {
Ok(Ok(())) => {
state.persist_metric_declaration_update(&name, &request).await;
state
.persist_metric_declaration_update(&name, &request)
.await;
let dto = MetricDeclarationDto {
name: request.name,
message_match_regex: request.message_match_regex,
@@ -486,7 +542,10 @@ async fn delete_metric(
if state
.engine
.tx
.send(EngineMessage::RemoveMetricDeclaration { name: name.clone(), respond_to })
.send(EngineMessage::RemoveMetricDeclaration {
name: name.clone(),
respond_to,
})
.await
.is_err()
{
@@ -552,7 +611,10 @@ async fn run_promql_query(
async fn list_themes() -> impl IntoResponse {
let mut names: Vec<String> = THEMES
.walk()
.filter_map(|file| file.file_name().map(|name| name.trim_end_matches(".css").to_string()))
.filter_map(|file| {
file.file_name()
.map(|name| name.trim_end_matches(".css").to_string())
})
.collect();
names.sort();
Json(names).into_response()
@@ -561,7 +623,11 @@ async fn list_themes() -> impl IntoResponse {
async fn get_theme_content(AxumPath(name): AxumPath<String>) -> impl IntoResponse {
// Themes are just filenames within a flat embedded dir, not filesystem paths, so reject
// anything that isn't a plain name to rule out path traversal.
if name.is_empty() || !name.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') {
if name.is_empty()
|| !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
{
return StatusCode::NOT_FOUND.into_response();
}
match THEMES.get_file(&format!("{name}.css")) {
@@ -577,7 +643,9 @@ impl AlertError {
fn into_response(self) -> axum::response::Response {
match self {
AlertError::NotFound => (StatusCode::NOT_FOUND, self.to_string()).into_response(),
AlertError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response(),
AlertError::Io(_) => {
(StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response()
}
}
}
}
@@ -592,7 +660,12 @@ async fn create_alert_rule(
Json(body): Json<CreateAlertRuleRequest>,
) -> impl IntoResponse {
let mut settings = state.settings.lock().unwrap();
match settings.create_alert_rule(&state.settings_path, body.name, body.enabled, body.condition) {
match settings.create_alert_rule(
&state.settings_path,
body.name,
body.enabled,
body.condition,
) {
Ok(rule) => (StatusCode::CREATED, Json(rule)).into_response(),
Err(err) => err.into_response(),
}
@@ -604,13 +677,22 @@ async fn update_alert_rule(
Json(body): Json<UpdateAlertRuleRequest>,
) -> impl IntoResponse {
let mut settings = state.settings.lock().unwrap();
match settings.update_alert_rule(&state.settings_path, &id, body.name, body.enabled, body.condition) {
match settings.update_alert_rule(
&state.settings_path,
&id,
body.name,
body.enabled,
body.condition,
) {
Ok(rule) => Json(rule).into_response(),
Err(err) => err.into_response(),
}
}
async fn delete_alert_rule(State(state): State<ApiState>, AxumPath(id): AxumPath<String>) -> impl IntoResponse {
async fn delete_alert_rule(
State(state): State<ApiState>,
AxumPath(id): AxumPath<String>,
) -> impl IntoResponse {
let mut settings = state.settings.lock().unwrap();
match settings.remove_alert_rule(&state.settings_path, &id) {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
@@ -618,7 +700,9 @@ async fn delete_alert_rule(State(state): State<ApiState>, AxumPath(id): AxumPath
}
}
async fn test_alert_notification(Json(body): Json<TestAlertNotificationRequest>) -> impl IntoResponse {
async fn test_alert_notification(
Json(body): Json<TestAlertNotificationRequest>,
) -> impl IntoResponse {
let server = body.server.unwrap_or_else(|| "https://ntfy.sh".to_string());
match deliver_ntfy(&server, &body.topic, "Test notification from ottyl").await {
Ok(()) => StatusCode::NO_CONTENT.into_response(),
@@ -652,7 +736,9 @@ async fn update_settings(
}
match settings.save(&state.settings_path) {
Ok(()) => (),
Err(err) => return (StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(),
Err(err) => {
return (StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response();
}
}
(SettingsDto::from(&*settings), settings.max_logs)
};
@@ -671,16 +757,26 @@ impl ProfileError {
fn into_response(self) -> axum::response::Response {
match self {
ProfileError::NotFound => (StatusCode::NOT_FOUND, self.to_string()).into_response(),
ProfileError::DuplicateName => (StatusCode::BAD_REQUEST, self.to_string()).into_response(),
ProfileError::CannotDeleteDefault => (StatusCode::BAD_REQUEST, self.to_string()).into_response(),
ProfileError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response(),
ProfileError::DuplicateName => {
(StatusCode::BAD_REQUEST, self.to_string()).into_response()
}
ProfileError::CannotDeleteDefault => {
(StatusCode::BAD_REQUEST, self.to_string()).into_response()
}
ProfileError::Io(_) => {
(StatusCode::INTERNAL_SERVER_ERROR, self.to_string()).into_response()
}
}
}
}
async fn list_profiles(State(state): State<ApiState>) -> impl IntoResponse {
let settings = state.settings.lock().unwrap();
let summaries: Vec<ProfileSummaryDto> = settings.profiles.iter().map(ProfileSummaryDto::from).collect();
let summaries: Vec<ProfileSummaryDto> = settings
.profiles
.iter()
.map(ProfileSummaryDto::from)
.collect();
Json(summaries).into_response()
}
@@ -691,7 +787,9 @@ async fn create_profile(
let mut settings = state.settings.lock().unwrap();
match settings.create_profile(&state.settings_path, body.name.clone()) {
Ok(()) => {
let profile = settings.profile(&body.name).expect("profile was just created by create_profile");
let profile = settings
.profile(&body.name)
.expect("profile was just created by create_profile");
(StatusCode::CREATED, Json(ProfileSummaryDto::from(profile))).into_response()
}
Err(err) => err.into_response(),
@@ -717,7 +815,10 @@ async fn reload_engine_declarations(engine: &EngineHandle, profile: &Profile) {
let (respond_to, response) = oneshot::channel();
if engine
.tx
.send(EngineMessage::RemoveParamDeclaration { name: param.name, respond_to })
.send(EngineMessage::RemoveParamDeclaration {
name: param.name,
respond_to,
})
.await
.is_ok()
{
@@ -732,7 +833,10 @@ async fn reload_engine_declarations(engine: &EngineHandle, profile: &Profile) {
let (respond_to, response) = oneshot::channel();
if engine
.tx
.send(EngineMessage::RemoveMetricDeclaration { name: metric.name, respond_to })
.send(EngineMessage::RemoveMetricDeclaration {
name: metric.name,
respond_to,
})
.await
.is_ok()
{
@@ -835,13 +939,13 @@ async fn delete_dashboard(
#[cfg(test)]
mod tests {
use super::*;
use crate::engine::{self, IngestedLine};
use crate::config::Settings;
use crate::engine::timestamp::Timestamp;
use crate::engine::{self, IngestedLine};
use axum::body::Body;
use axum::http::Request;
use http_body_util::BodyExt;
use serde_json::{json, Value};
use serde_json::{Value, json};
use std::sync::Mutex;
use tower::ServiceExt;
@@ -875,7 +979,11 @@ mod tests {
}
async fn get_json(app: &Router, uri: &str) -> (StatusCode, Value) {
let request = Request::builder().method("GET").uri(uri).body(Body::empty()).unwrap();
let request = Request::builder()
.method("GET")
.uri(uri)
.body(Body::empty())
.unwrap();
let response = app.clone().oneshot(request).await.unwrap();
let status = response.status();
let bytes = response.into_body().collect().await.unwrap().to_bytes();
@@ -982,11 +1090,22 @@ mod tests {
let settings_path = dir.path().join("settings.json");
{
let mut settings = settings_handle.lock().unwrap();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
}
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
declare_test_param(
&app,
"path",
r"request path=(?P<path>\S+)",
"path",
"string",
)
.await;
ingest_lines(&engine, 0, &["request path=/a", "level=info x=1"]).await;
let (_, before) = get_json(&app, "/api/params").await;
@@ -1008,8 +1127,12 @@ mod tests {
assert_eq!(result["logs_considered"], 2);
let (_, after) = get_json(&app, "/api/params").await;
let after_declared: Vec<_> =
after.as_array().unwrap().iter().filter(|p| p["reserved"] != true).collect();
let after_declared: Vec<_> = after
.as_array()
.unwrap()
.iter()
.filter(|p| p["reserved"] != true)
.collect();
assert_eq!(after_declared.len(), 3);
assert!(after_declared.iter().all(|p| p["auto_derived"] == true));
@@ -1037,7 +1160,13 @@ mod tests {
assert!(body["error"].as_str().unwrap().contains("invalid regex"));
}
async fn declare_test_param(app: &Router, name: &str, pattern: &str, capture_group: &str, kind: &str) {
async fn declare_test_param(
app: &Router,
name: &str,
pattern: &str,
capture_group: &str,
kind: &str,
) {
let (status, _body) = send_json(
app,
"POST",
@@ -1056,7 +1185,14 @@ mod tests {
#[tokio::test]
async fn declares_metric_and_shows_up_in_list() {
let (app, _engine, _settings, _dir) = test_app();
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
declare_test_param(
&app,
"path",
r"request path=(?P<path>\S+)",
"path",
"string",
)
.await;
let (status, body) = send_json(
&app,
"POST",
@@ -1099,8 +1235,22 @@ mod tests {
#[tokio::test]
async fn updates_metric_declaration_in_place_rather_than_duplicating() {
let (app, _engine, _settings, _dir) = test_app();
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
declare_test_param(&app, "duration", r"duration=(?P<duration>\d+)", "duration", "number").await;
declare_test_param(
&app,
"path",
r"request path=(?P<path>\S+)",
"path",
"string",
)
.await;
declare_test_param(
&app,
"duration",
r"duration=(?P<duration>\d+)",
"duration",
"number",
)
.await;
send_json(
&app,
"POST",
@@ -1220,8 +1370,12 @@ mod tests {
let settings_path = dir.path().join("settings.json");
{
let mut settings = settings_handle.lock().unwrap();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
}
send_json(
@@ -1235,19 +1389,39 @@ mod tests {
}),
)
.await;
assert_eq!(Settings::load(&settings_path).profile("web").unwrap().metrics.len(), 1);
assert_eq!(
Settings::load(&settings_path)
.profile("web")
.unwrap()
.metrics
.len(),
1
);
let status = delete_request(&app, "/api/metrics/requests").await;
assert_eq!(status, StatusCode::NO_CONTENT);
assert!(Settings::load(&settings_path).profile("web").unwrap().metrics.is_empty());
assert!(
Settings::load(&settings_path)
.profile("web")
.unwrap()
.metrics
.is_empty()
);
}
#[tokio::test]
async fn promql_query_against_deleted_metric_fails_with_readable_error() {
let (app, engine, _settings, _dir) = test_app();
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
declare_test_param(
&app,
"path",
r"request path=(?P<path>\S+)",
"path",
"string",
)
.await;
send_json(
&app,
"POST",
@@ -1288,7 +1462,14 @@ mod tests {
async fn queries_bucket_values_with_label_filter_and_range() {
let (app, engine, _settings, _dir) = test_app();
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
declare_test_param(
&app,
"path",
r"request path=(?P<path>\S+)",
"path",
"string",
)
.await;
send_json(
&app,
"POST",
@@ -1341,7 +1522,14 @@ mod tests {
async fn promql_query_returns_series_for_seeded_fixture() {
let (app, engine, _settings, _dir) = test_app();
declare_test_param(&app, "path", r"request path=(?P<path>\S+)", "path", "string").await;
declare_test_param(
&app,
"path",
r"request path=(?P<path>\S+)",
"path",
"string",
)
.await;
send_json(
&app,
"POST",
@@ -1398,8 +1586,12 @@ mod tests {
let settings_path = dir.path().join("settings.json");
{
let mut settings = settings_handle.lock().unwrap();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
}
let (status, _body) = send_json(
@@ -1429,8 +1621,12 @@ mod tests {
let settings_path = dir.path().join("settings.json");
{
let mut settings = settings_handle.lock().unwrap();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
}
ingest_lines(&engine, 0, &["dx=1 moved"]).await;
@@ -1459,7 +1655,10 @@ mod tests {
let fresh_engine_tx = crate::engine::spawn(fresh_settings_handle, 16);
profile.load_into_engine(&fresh_engine_tx).await;
let response = fresh_engine_tx.ask(EngineQuery::MetricDeclarations).await.unwrap();
let response = fresh_engine_tx
.ask(EngineQuery::MetricDeclarations)
.await
.unwrap();
let EngineResponse::MetricDeclarations(metrics) = response else {
panic!("unexpected response");
};
@@ -1468,12 +1667,20 @@ mod tests {
}
async fn delete_request(app: &Router, uri: &str) -> StatusCode {
let request = Request::builder().method("DELETE").uri(uri).body(Body::empty()).unwrap();
let request = Request::builder()
.method("DELETE")
.uri(uri)
.body(Body::empty())
.unwrap();
app.clone().oneshot(request).await.unwrap().status()
}
async fn get_status(app: &Router, uri: &str) -> StatusCode {
let request = Request::builder().method("GET").uri(uri).body(Body::empty()).unwrap();
let request = Request::builder()
.method("GET")
.uri(uri)
.body(Body::empty())
.unwrap();
app.clone().oneshot(request).await.unwrap().status()
}
@@ -1481,7 +1688,13 @@ mod tests {
async fn dashboard_crud_round_trips_through_http() {
let (app, _engine, _settings, _dir) = test_app();
let (status, _body) = send_json(&app, "POST", "/api/settings/profiles", json!({ "name": "web" })).await;
let (status, _body) = send_json(
&app,
"POST",
"/api/settings/profiles",
json!({ "name": "web" }),
)
.await;
assert_eq!(status, StatusCode::CREATED);
let (status, body) = send_json(
@@ -1518,12 +1731,19 @@ mod tests {
assert_eq!(panels.len(), 1);
assert_eq!(panels[0]["chart_type"], "line");
let (status, body) =
get_json(&app, &format!("/api/settings/profiles/web/dashboards/{dashboard_id}")).await;
let (status, body) = get_json(
&app,
&format!("/api/settings/profiles/web/dashboards/{dashboard_id}"),
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body["panels"].as_array().unwrap().len(), 1);
let status = delete_request(&app, &format!("/api/settings/profiles/web/dashboards/{dashboard_id}")).await;
let status = delete_request(
&app,
&format!("/api/settings/profiles/web/dashboards/{dashboard_id}"),
)
.await;
assert_eq!(status, StatusCode::NO_CONTENT);
let (status, body) = get_json(&app, "/api/settings/profiles/web/dashboards").await;
assert_eq!(status, StatusCode::OK);
@@ -1536,7 +1756,13 @@ mod tests {
let status = get_status(&app, "/api/settings/profiles/ghost/dashboards").await;
assert_eq!(status, StatusCode::NOT_FOUND);
let (status, _body) = send_json(&app, "POST", "/api/settings/profiles", json!({ "name": "web" })).await;
let (status, _body) = send_json(
&app,
"POST",
"/api/settings/profiles",
json!({ "name": "web" }),
)
.await;
assert_eq!(status, StatusCode::CREATED);
let status = get_status(&app, "/api/settings/profiles/web/dashboards/does-not-exist").await;
assert_eq!(status, StatusCode::NOT_FOUND);
@@ -1546,7 +1772,13 @@ mod tests {
async fn saved_dashboard_persists_across_simulated_restart() {
let (app, _engine, _settings_handle, dir) = test_app();
let (status, _body) = send_json(&app, "POST", "/api/settings/profiles", json!({ "name": "web" })).await;
let (status, _body) = send_json(
&app,
"POST",
"/api/settings/profiles",
json!({ "name": "web" }),
)
.await;
assert_eq!(status, StatusCode::CREATED);
let (status, body) = send_json(
&app,
@@ -1605,7 +1837,8 @@ mod tests {
assert_eq!(status, StatusCode::OK);
assert_eq!(body.as_array().unwrap().len(), 3);
let (status, body) = send_json(&app, "PATCH", "/api/settings", json!({ "max_logs": 2 })).await;
let (status, body) =
send_json(&app, "PATCH", "/api/settings", json!({ "max_logs": 2 })).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body["max_logs"], 2);
@@ -1644,7 +1877,8 @@ mod tests {
let (_, body) = get_json(&app, "/api/logs").await;
assert_eq!(body.as_array().unwrap().len(), 1);
let (status, body) = send_json(&app, "PATCH", "/api/settings", json!({ "max_logs": null })).await;
let (status, body) =
send_json(&app, "PATCH", "/api/settings", json!({ "max_logs": null })).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body["max_logs"], Value::Null);
@@ -1657,7 +1891,14 @@ mod tests {
async fn get_logs_sort_by_orders_by_param_and_toggles_missing_param_handling() {
let (app, engine, _settings, _dir) = test_app();
declare_test_param(&app, "priority", r"priority=(?P<priority>\d+)", "priority", "number").await;
declare_test_param(
&app,
"priority",
r"priority=(?P<priority>\d+)",
"priority",
"number",
)
.await;
ingest_lines(
&engine,
@@ -1673,17 +1914,36 @@ mod tests {
let (status, body) = get_json(&app, "/api/logs").await;
assert_eq!(status, StatusCode::OK);
let sequences: Vec<u64> = body.as_array().unwrap().iter().map(|e| e["sequence"].as_u64().unwrap()).collect();
let sequences: Vec<u64> = body
.as_array()
.unwrap()
.iter()
.map(|e| e["sequence"].as_u64().unwrap())
.collect();
assert_eq!(sequences, vec![0, 1, 2, 3]);
let (status, body) = get_json(&app, "/api/logs?sort_by=priority").await;
assert_eq!(status, StatusCode::OK);
let sequences: Vec<u64> = body.as_array().unwrap().iter().map(|e| e["sequence"].as_u64().unwrap()).collect();
let sequences: Vec<u64> = body
.as_array()
.unwrap()
.iter()
.map(|e| e["sequence"].as_u64().unwrap())
.collect();
assert_eq!(sequences, vec![2, 0, 1, 3]);
let (status, body) = get_json(&app, "/api/logs?sort_by=priority&sort_missing_fallback=false").await;
let (status, body) = get_json(
&app,
"/api/logs?sort_by=priority&sort_missing_fallback=false",
)
.await;
assert_eq!(status, StatusCode::OK);
let sequences: Vec<u64> = body.as_array().unwrap().iter().map(|e| e["sequence"].as_u64().unwrap()).collect();
let sequences: Vec<u64> = body
.as_array()
.unwrap()
.iter()
.map(|e| e["sequence"].as_u64().unwrap())
.collect();
assert_eq!(sequences, vec![2, 0]);
}
src/cli/mod.rs
+13 -6
View File
@@ -86,15 +86,16 @@ pub async fn run() {
register_session_and_log(&settings_path, &args, bound_port);
axum::serve(listener, app)
.await
.expect("server error");
axum::serve(listener, app).await.expect("server error");
}
async fn spawn_engine(settings: &Arc<Mutex<Settings>>) -> crate::engine::EngineHandle {
let engine_tx = crate::engine::spawn(settings.clone(), crate::consts::ENGINE_CHANNEL_CAPACITY);
let max_logs = settings.lock().unwrap().max_logs;
let _ = engine_tx.tx.send(crate::engine::EngineMessage::SetMaxLogs(max_logs)).await;
let _ = engine_tx
.tx
.send(crate::engine::EngineMessage::SetMaxLogs(max_logs))
.await;
engine_tx
}
@@ -142,7 +143,10 @@ fn spawn_ingest(args: &Args, engine_tx: &crate::engine::EngineHandle) {
}
if let Some(metrics_url) = args.metrics_url.clone() {
tokio::spawn(crate::engine::ingest::run_metrics_poll(engine_tx.clone(), metrics_url));
tokio::spawn(crate::engine::ingest::run_metrics_poll(
engine_tx.clone(),
metrics_url,
));
}
}
@@ -198,7 +202,10 @@ fn print_sessions(settings_path: &std::path::Path) {
return;
}
for session in sessions {
println!("{}\tport={}\tpid={}", session.name, session.port, session.pid);
println!(
"{}\tport={}\tpid={}",
session.name, session.port, session.pid
);
}
}
src/cli/sessions.rs
+5 -1
View File
@@ -131,7 +131,11 @@ mod tests {
let sessions_dir_path = sessions_dir(&settings_path);
fs::create_dir_all(&sessions_dir_path).unwrap();
let stale = SessionInfo { name: "foo".to_string(), port: 1111, pid: 999_999 };
let stale = SessionInfo {
name: "foo".to_string(),
port: 1111,
pid: 999_999,
};
fs::write(
session_file_path(&sessions_dir_path, "foo"),
serde_json::to_string(&stale).unwrap(),
src/config/mod.rs
+10 -7
View File
@@ -8,8 +8,8 @@ use std::sync::{Arc, Mutex};
use directories::ProjectDirs;
use serde::{Deserialize, Serialize};
use crate::engine::alerts::AlertRule;
use crate::config::profile::Profile;
use crate::engine::alerts::AlertRule;
pub type SettingsHandle = Arc<Mutex<Settings>>;
@@ -70,13 +70,13 @@ impl From<SettingsOnDisk> for Settings {
ProfilesOnDisk::Inline(profiles) => (profiles, None),
ProfilesOnDisk::Legacy(pointer) => (Vec::new(), pointer.active_profile),
};
let theme = on_disk.theme.unwrap_or_else(|| {
match on_disk.color_scheme.as_deref() {
let theme = on_disk
.theme
.unwrap_or_else(|| match on_disk.color_scheme.as_deref() {
Some("grey") => "dark".to_string(),
Some(scheme) => scheme.to_string(),
None => default_theme(),
}
});
});
Settings {
theme,
profiles,
@@ -185,7 +185,8 @@ impl Settings {
.map(|parent| parent.join("profiles"))
.unwrap_or_else(|| PathBuf::from("profiles"))
.join("profiles.json");
let Some(profiles) = crate::config::profile::Profile::load_legacy_store(&legacy_path) else {
let Some(profiles) = crate::config::profile::Profile::load_legacy_store(&legacy_path)
else {
return;
};
self.profiles = profiles;
@@ -271,7 +272,9 @@ mod tests {
theme: "dark".to_string(),
..Settings::default()
};
settings.profiles.push(crate::config::profile::Profile::new("default".to_string()));
settings
.profiles
.push(crate::config::profile::Profile::new("default".to_string()));
settings.active_profile = Some("default".to_string());
settings.alert_rules.push(crate::engine::alerts::AlertRule {
id: "rule1".to_string(),
src/config/profile.rs
+297 -92
View File
@@ -1,7 +1,7 @@
#![allow(dead_code)]
use std::collections::hash_map::DefaultHasher;
use std::collections::BTreeMap;
use std::collections::hash_map::DefaultHasher;
use std::fmt;
use std::fs;
use std::hash::{Hash, Hasher};
@@ -11,10 +11,10 @@ use std::time::{SystemTime, UNIX_EPOCH};
use serde::{Deserialize, Serialize};
use crate::config::Settings;
use crate::consts::{DEFAULT_PROFILE_NAME, SEEDED_METRIC_LABEL, SEEDED_METRIC_NAME};
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::{DeclareMetricRequest, DeclareParamRequest, EngineHandle, EngineMessage};
use crate::config::Settings;
use tokio::sync::oneshot;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
@@ -167,8 +167,11 @@ impl Profile {
let mut label_params = Vec::new();
let mut value_param = None;
for matcher in &legacy {
let param_name =
self.resolve_or_create_legacy_param(&metric_name, &message_match_regex, matcher);
let param_name = self.resolve_or_create_legacy_param(
&metric_name,
&message_match_regex,
matcher,
);
match matcher.role {
LegacyParamRole::Label => label_params.push(param_name),
LegacyParamRole::Value => value_param = Some(param_name),
@@ -187,14 +190,13 @@ impl Profile {
message_match_regex: &str,
matcher: &LegacyMetricParamRecord,
) -> String {
if let Some(existing) = self.params.iter().find(|p| p.name == matcher.param_name) {
if existing.message_match_regex == message_match_regex
&& existing.capture_group == matcher.capture_group
&& existing.json_field_path == matcher.json_field_path
&& existing.kind == matcher.kind
{
return existing.name.clone();
}
if let Some(existing) = self.params.iter().find(|p| p.name == matcher.param_name)
&& existing.message_match_regex == message_match_regex
&& existing.capture_group == matcher.capture_group
&& existing.json_field_path == matcher.json_field_path
&& existing.kind == matcher.kind
{
return existing.name.clone();
}
let implicit_name = format!("{metric_name}.{}", matcher.param_name);
@@ -213,8 +215,11 @@ impl Profile {
}
fn migrate_duration_value_params(&mut self) -> bool {
let value_param_names: std::collections::HashSet<&str> =
self.metrics.iter().filter_map(|metric| metric.value_param.as_deref()).collect();
let value_param_names: std::collections::HashSet<&str> = self
.metrics
.iter()
.filter_map(|metric| metric.value_param.as_deref())
.collect();
let mut migrated = false;
for param in &mut self.params {
if !param.elapsed_since_last
@@ -267,7 +272,11 @@ fn generate_id(seed: &str) -> String {
impl Dashboard {
pub fn new(name: String) -> Dashboard {
let id = generate_id(&name);
Dashboard { id, name, panels: Vec::new() }
Dashboard {
id,
name,
panels: Vec::new(),
}
}
}
@@ -278,8 +287,14 @@ impl MetricDeclarationRecord {
message_match_regex: r"(?i)error".to_string(),
label_params: Vec::new(),
value_param: None,
matched_labels: Some(BTreeMap::from([(SEEDED_METRIC_LABEL.to_string(), "error".to_string())])),
unmatched_labels: Some(BTreeMap::from([(SEEDED_METRIC_LABEL.to_string(), "other".to_string())])),
matched_labels: Some(BTreeMap::from([(
SEEDED_METRIC_LABEL.to_string(),
"error".to_string(),
)])),
unmatched_labels: Some(BTreeMap::from([(
SEEDED_METRIC_LABEL.to_string(),
"other".to_string(),
)])),
legacy_params: Vec::new(),
}
}
@@ -336,11 +351,17 @@ impl Profile {
let name = request.name.clone();
if engine
.tx
.send(EngineMessage::DeclareParam { request, respond_to })
.send(EngineMessage::DeclareParam {
request,
respond_to,
})
.await
.is_err()
{
eprintln!("warning: engine unavailable while loading profile '{}'", self.name);
eprintln!(
"warning: engine unavailable while loading profile '{}'",
self.name
);
return;
}
match response.await {
@@ -362,11 +383,17 @@ impl Profile {
let name = request.name.clone();
if engine
.tx
.send(EngineMessage::DeclareMetric { request, respond_to })
.send(EngineMessage::DeclareMetric {
request,
respond_to,
})
.await
.is_err()
{
eprintln!("warning: engine unavailable while loading profile '{}'", self.name);
eprintln!(
"warning: engine unavailable while loading profile '{}'",
self.name
);
return;
}
match response.await {
@@ -444,7 +471,11 @@ impl Settings {
self.profiles.iter().find(|p| p.name == name)
}
pub fn create_profile(&mut self, settings_path: &Path, name: String) -> Result<(), ProfileError> {
pub fn create_profile(
&mut self,
settings_path: &Path,
name: String,
) -> Result<(), ProfileError> {
if self.profile(&name).is_some() {
return Err(ProfileError::DuplicateName);
}
@@ -463,7 +494,11 @@ impl Settings {
if name == DEFAULT_PROFILE_NAME {
return Err(ProfileError::CannotDeleteDefault);
}
let index = self.profiles.iter().position(|p| p.name == name).ok_or(ProfileError::NotFound)?;
let index = self
.profiles
.iter()
.position(|p| p.name == name)
.ok_or(ProfileError::NotFound)?;
self.profiles.remove(index);
if self.active_profile.as_deref() == Some(name) {
self.active_profile = None;
@@ -475,7 +510,9 @@ impl Settings {
pub fn ensure_default_profile(&mut self, settings_path: &Path) -> Result<(), ProfileError> {
if self.profile(DEFAULT_PROFILE_NAME).is_none() {
let mut profile = Profile::new(DEFAULT_PROFILE_NAME.to_string());
profile.metrics.push(MetricDeclarationRecord::seeded_error_split());
profile
.metrics
.push(MetricDeclarationRecord::seeded_error_split());
profile.dashboards.push(Dashboard::seeded_default());
self.profiles.push(profile);
}
@@ -491,10 +528,10 @@ impl Settings {
settings_path: &Path,
name: Option<String>,
) -> Result<(), ProfileError> {
if let Some(name) = &name {
if self.profile(name).is_none() {
return Err(ProfileError::NotFound);
}
if let Some(name) = &name
&& self.profile(name).is_none()
{
return Err(ProfileError::NotFound);
}
self.active_profile = name;
self.save(settings_path)?;
@@ -678,10 +715,10 @@ impl Settings {
#[cfg(test)]
mod tests {
use super::*;
use crate::engine::data_value::MetricPrimitiveKind;
use crate::api::dto::LogsParams;
use crate::engine::{EngineMessage, EngineQuery, EngineResponse, IngestedLine};
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::timestamp::Timestamp;
use crate::engine::{EngineMessage, EngineQuery, EngineResponse, IngestedLine};
fn param_request(name: &str, pattern: &str, capture_group: &str) -> DeclareParamRequest {
DeclareParamRequest {
@@ -719,8 +756,12 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.create_profile(&settings_path, "worker".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.create_profile(&settings_path, "worker".to_string())
.unwrap();
assert_eq!(settings.profiles.len(), 2);
assert!(settings.profile("web").is_some());
@@ -750,13 +791,27 @@ mod tests {
settings.ensure_default_profile(&settings_path).unwrap();
assert!(settings.profile(DEFAULT_PROFILE_NAME).is_some());
assert_eq!(settings.active_profile, Some(DEFAULT_PROFILE_NAME.to_string()));
assert_eq!(
settings.active_profile,
Some(DEFAULT_PROFILE_NAME.to_string())
);
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
settings.ensure_default_profile(&settings_path).unwrap();
assert_eq!(settings.profiles.iter().filter(|p| p.name == DEFAULT_PROFILE_NAME).count(), 1);
assert_eq!(
settings
.profiles
.iter()
.filter(|p| p.name == DEFAULT_PROFILE_NAME)
.count(),
1
);
assert_eq!(settings.active_profile, Some("web".to_string()));
}
@@ -773,20 +828,33 @@ mod tests {
assert_eq!(default_profile.metrics[0].name, SEEDED_METRIC_NAME);
assert_eq!(
default_profile.metrics[0].matched_labels,
Some(BTreeMap::from([(SEEDED_METRIC_LABEL.to_string(), "error".to_string())]))
Some(BTreeMap::from([(
SEEDED_METRIC_LABEL.to_string(),
"error".to_string()
)]))
);
assert_eq!(
default_profile.metrics[0].unmatched_labels,
Some(BTreeMap::from([(SEEDED_METRIC_LABEL.to_string(), "other".to_string())]))
Some(BTreeMap::from([(
SEEDED_METRIC_LABEL.to_string(),
"other".to_string()
)]))
);
assert_eq!(default_profile.dashboards.len(), 1);
let dashboard = &default_profile.dashboards[0];
assert_eq!(dashboard.panels.len(), 2);
assert!(dashboard
.panels
.iter()
.any(|p| p.chart_type == ChartType::Line && p.promql.contains(SEEDED_METRIC_NAME)));
assert!(dashboard.panels.iter().any(|p| p.chart_type == ChartType::Pie && p.promql == SEEDED_METRIC_NAME));
assert!(
dashboard
.panels
.iter()
.any(|p| p.chart_type == ChartType::Line && p.promql.contains(SEEDED_METRIC_NAME))
);
assert!(
dashboard
.panels
.iter()
.any(|p| p.chart_type == ChartType::Pie && p.promql == SEEDED_METRIC_NAME)
);
let mut reloaded = Settings::load(&settings_path);
reloaded.ensure_default_profile(&settings_path).unwrap();
@@ -797,8 +865,8 @@ mod tests {
#[tokio::test]
async fn seeded_metric_renders_real_error_vs_other_data() {
use crate::engine::{self, EngineMessage, EngineQuery, EngineResponse, IngestedLine};
use crate::engine::timestamp::Timestamp;
use crate::engine::{self, EngineMessage, EngineQuery, EngineResponse, IngestedLine};
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
@@ -810,7 +878,11 @@ mod tests {
let engine_tx = engine::spawn(settings_handle, 128);
default_profile.load_into_engine(&engine_tx).await;
for (sequence, text) in [(0u64, "all good"), (1, "an ERROR happened"), (2, "fine again")] {
for (sequence, text) in [
(0u64, "all good"),
(1, "an ERROR happened"),
(2, "fine again"),
] {
engine_tx
.tx
.send(EngineMessage::Ingest(IngestedLine {
@@ -825,18 +897,38 @@ mod tests {
let response = ask(
&engine_tx,
EngineQuery::Promql { query: SEEDED_METRIC_NAME.to_string(), start_ms: None, end_ms: None },
EngineQuery::Promql {
query: SEEDED_METRIC_NAME.to_string(),
start_ms: None,
end_ms: None,
},
)
.await;
let EngineResponse::Promql(Ok(mut series)) = response else {
panic!("expected a successful Promql response");
};
series.sort_by(|a, b| a.labels.get(SEEDED_METRIC_LABEL).cmp(&b.labels.get(SEEDED_METRIC_LABEL)));
series.sort_by(|a, b| {
a.labels
.get(SEEDED_METRIC_LABEL)
.cmp(&b.labels.get(SEEDED_METRIC_LABEL))
});
assert_eq!(series.len(), 2);
assert_eq!(series[0].labels.get(SEEDED_METRIC_LABEL).map(String::as_str), Some("error"));
assert_eq!(
series[0]
.labels
.get(SEEDED_METRIC_LABEL)
.map(String::as_str),
Some("error")
);
assert_eq!(series[0].points.len(), 1);
assert_eq!(series[1].labels.get(SEEDED_METRIC_LABEL).map(String::as_str), Some("other"));
assert_eq!(
series[1]
.labels
.get(SEEDED_METRIC_LABEL)
.map(String::as_str),
Some("other")
);
assert_eq!(series[1].points.len(), 2);
}
@@ -859,19 +951,31 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
let param_req = param_request("level", r"level=(?P<level>\w+)", "level");
settings.record_param_declaration(&settings_path, "web", &param_req).unwrap();
settings.create_dashboard(&settings_path, "web", "Overview".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.record_param_declaration(&settings_path, "web", &param_req)
.unwrap();
settings
.create_dashboard(&settings_path, "web", "Overview".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
settings.create_profile(&settings_path, "web-copy".to_string()).unwrap();
settings
.create_profile(&settings_path, "web-copy".to_string())
.unwrap();
let copy = settings.profile("web-copy").unwrap().clone();
assert_eq!(copy.params, settings.profile("web").unwrap().params);
assert_eq!(copy.dashboards, settings.profile("web").unwrap().dashboards);
settings.set_active_profile(&settings_path, None).unwrap();
settings.create_profile(&settings_path, "blank".to_string()).unwrap();
settings
.create_profile(&settings_path, "blank".to_string())
.unwrap();
let blank = settings.profile("blank").unwrap();
assert!(blank.params.is_empty());
assert!(blank.dashboards.is_empty());
@@ -883,13 +987,21 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.create_profile(&settings_path, "worker".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.create_profile(&settings_path, "worker".to_string())
.unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
assert_eq!(settings.active_profile, Some("web".to_string()));
settings.set_active_profile(&settings_path, Some("worker".to_string())).unwrap();
settings
.set_active_profile(&settings_path, Some("worker".to_string()))
.unwrap();
let reloaded = Settings::load(&settings_path);
assert_eq!(reloaded.active_profile, Some("worker".to_string()));
@@ -907,13 +1019,19 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
let param_req = param_request("level", r"level=(?P<level>\w+)", "level");
settings.record_param_declaration(&settings_path, "web", &param_req).unwrap();
settings
.record_param_declaration(&settings_path, "web", &param_req)
.unwrap();
let metric_req = metric_request("requests", r"duration=(?P<duration>\d+)", "duration");
settings.record_metric_declaration(&settings_path, "web", &metric_req, &[]).unwrap();
settings
.record_metric_declaration(&settings_path, "web", &metric_req, &[])
.unwrap();
let profile = settings.profile("web").unwrap();
assert_eq!(profile.params.len(), 1);
@@ -922,7 +1040,9 @@ mod tests {
assert_eq!(profile.metrics[0].name, "requests");
let param_req_v2 = param_request("level", r"lvl=(?P<level>\w+)", "level");
settings.record_param_declaration(&settings_path, "web", &param_req_v2).unwrap();
settings
.record_param_declaration(&settings_path, "web", &param_req_v2)
.unwrap();
let profile = settings.profile("web").unwrap();
assert_eq!(profile.params.len(), 1);
assert_eq!(profile.params[0].message_match_regex, r"lvl=(?P<level>\w+)");
@@ -939,15 +1059,23 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
let metric_req = metric_request("requests", r"duration=(?P<duration>\d+)", "duration");
settings.record_metric_declaration(&settings_path, "web", &metric_req, &[]).unwrap();
settings
.record_metric_declaration(&settings_path, "web", &metric_req, &[])
.unwrap();
assert_eq!(settings.profile("web").unwrap().metrics.len(), 1);
settings.remove_metric_declaration(&settings_path, "web", "requests").unwrap();
settings
.remove_metric_declaration(&settings_path, "web", "requests")
.unwrap();
assert!(settings.profile("web").unwrap().metrics.is_empty());
settings.remove_metric_declaration(&settings_path, "web", "never-declared").unwrap();
settings
.remove_metric_declaration(&settings_path, "web", "never-declared")
.unwrap();
let reloaded = Settings::load(&settings_path);
assert!(reloaded.profile("web").unwrap().metrics.is_empty());
@@ -959,7 +1087,9 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.record_param_declaration(
&settings_path,
@@ -971,18 +1101,28 @@ mod tests {
.record_param_declaration(
&settings_path,
"web",
&param_request("duration", r"request duration=(?P<duration>\d+)", "duration"),
&param_request(
"duration",
r"request duration=(?P<duration>\d+)",
"duration",
),
)
.unwrap();
settings
.record_metric_declaration(
&settings_path,
"web",
&metric_request("requests", r"request duration=(?P<duration>\d+)", "duration"),
&metric_request(
"requests",
r"request duration=(?P<duration>\d+)",
"duration",
),
&[],
)
.unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
drop(settings);
let settings = Settings::load(&settings_path);
@@ -1020,7 +1160,10 @@ mod tests {
};
assert_eq!(metrics.len(), 1);
assert_eq!(metrics[0].name, "requests");
assert_eq!(metrics[0].message_match_regex, r"request duration=(?P<duration>\d+)");
assert_eq!(
metrics[0].message_match_regex,
r"request duration=(?P<duration>\d+)"
);
assert_eq!(metrics[0].label_params, vec!["duration".to_string()]);
let logs = match ask(
@@ -1050,9 +1193,13 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
let dashboard = settings.create_dashboard(&settings_path, "web", "Overview".to_string()).unwrap();
let dashboard = settings
.create_dashboard(&settings_path, "web", "Overview".to_string())
.unwrap();
assert_eq!(dashboard.name, "Overview");
assert!(dashboard.panels.is_empty());
@@ -1077,7 +1224,9 @@ mod tests {
assert_eq!(dashboards[0].panels.len(), 1);
assert_eq!(dashboards[0].panels[0].promql, r#"requests{path="/a"}"#);
reloaded.delete_dashboard(&settings_path, "web", &dashboard.id).unwrap();
reloaded
.delete_dashboard(&settings_path, "web", &dashboard.id)
.unwrap();
assert!(reloaded.list_dashboards("web").unwrap().is_empty());
let reloaded_again = Settings::load(&settings_path);
assert!(reloaded_again.list_dashboards("web").unwrap().is_empty());
@@ -1125,16 +1274,39 @@ mod tests {
let profile = settings.profile("legacy").unwrap();
assert_eq!(profile.params.len(), 2);
let path_param = profile.params.iter().find(|p| p.name == "requests.path").unwrap();
assert_eq!(path_param.message_match_regex, profile.metrics[0].message_match_regex);
let path_param = profile
.params
.iter()
.find(|p| p.name == "requests.path")
.unwrap();
assert_eq!(
path_param.message_match_regex,
profile.metrics[0].message_match_regex
);
assert_eq!(path_param.capture_group, "path");
assert!(matches!(path_param.kind, ProfileMetricPrimitiveKind::String));
let duration_param = profile.params.iter().find(|p| p.name == "requests.duration").unwrap();
assert!(matches!(duration_param.kind, ProfileMetricPrimitiveKind::Number));
assert!(matches!(
path_param.kind,
ProfileMetricPrimitiveKind::String
));
let duration_param = profile
.params
.iter()
.find(|p| p.name == "requests.duration")
.unwrap();
assert!(matches!(
duration_param.kind,
ProfileMetricPrimitiveKind::Number
));
assert_eq!(profile.metrics.len(), 1);
assert_eq!(profile.metrics[0].label_params, vec!["requests.path".to_string()]);
assert_eq!(profile.metrics[0].value_param, Some("requests.duration".to_string()));
assert_eq!(
profile.metrics[0].label_params,
vec!["requests.path".to_string()]
);
assert_eq!(
profile.metrics[0].value_param,
Some("requests.duration".to_string())
);
let reloaded_raw = fs::read_to_string(&settings_path).unwrap();
assert!(!reloaded_raw.contains("param_name"));
@@ -1192,10 +1364,20 @@ mod tests {
let settings = Settings::load(&settings_path);
let profile = settings.profile("legacy").unwrap();
let gap_param = profile.params.iter().find(|p| p.name == "login_gap.gap").unwrap();
assert!(matches!(gap_param.kind, ProfileMetricPrimitiveKind::Duration));
let gap_param = profile
.params
.iter()
.find(|p| p.name == "login_gap.gap")
.unwrap();
assert!(matches!(
gap_param.kind,
ProfileMetricPrimitiveKind::Duration
));
assert!(gap_param.elapsed_since_last);
assert_eq!(profile.metrics[0].value_param, Some("login_gap.gap".to_string()));
assert_eq!(
profile.metrics[0].value_param,
Some("login_gap.gap".to_string())
);
}
#[test]
@@ -1229,7 +1411,14 @@ mod tests {
let gap_param = profile.params.iter().find(|p| p.name == "gap").unwrap();
assert!(gap_param.elapsed_since_last);
assert!(!profile.params.iter().find(|p| p.name == "user").unwrap().elapsed_since_last);
assert!(
!profile
.params
.iter()
.find(|p| p.name == "user")
.unwrap()
.elapsed_since_last
);
}
#[test]
@@ -1271,7 +1460,9 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.record_param_declaration(
&settings_path,
@@ -1280,7 +1471,12 @@ mod tests {
)
.unwrap();
settings
.record_metric_declaration(&settings_path, "web", &metric_request("requests", r"path=(?P<path>\S+)", "path"), &[])
.record_metric_declaration(
&settings_path,
"web",
&metric_request("requests", r"path=(?P<path>\S+)", "path"),
&[],
)
.unwrap();
let before = settings.profile("web").unwrap().clone();
@@ -1295,8 +1491,12 @@ mod tests {
let settings_path = dir.path().join("settings.json");
let mut settings = Settings::default();
settings.create_profile(&settings_path, "web".to_string()).unwrap();
settings.set_active_profile(&settings_path, Some("web".to_string())).unwrap();
settings
.create_profile(&settings_path, "web".to_string())
.unwrap();
settings
.set_active_profile(&settings_path, Some("web".to_string()))
.unwrap();
let settings_handle = std::sync::Arc::new(std::sync::Mutex::new(settings.clone()));
let engine_tx = crate::engine::spawn(settings_handle, 16);
@@ -1314,7 +1514,10 @@ mod tests {
let (respond_to, response) = oneshot::channel();
engine_tx
.tx
.send(EngineMessage::DeclareMetric { request: metric_req.clone(), respond_to })
.send(EngineMessage::DeclareMetric {
request: metric_req.clone(),
respond_to,
})
.await
.unwrap();
response.await.unwrap().unwrap();
@@ -1334,7 +1537,9 @@ mod tests {
kind: dx.kind.into(),
elapsed_since_last: dx.elapsed_since_last,
};
settings.record_metric_declaration(&settings_path, "web", &metric_req, &[promoted]).unwrap();
settings
.record_metric_declaration(&settings_path, "web", &metric_req, &[promoted])
.unwrap();
drop(settings);
let reloaded_settings = Settings::load(&settings_path);
src/engine/alerts.rs
+250 -66
View File
@@ -9,12 +9,12 @@ use std::time::{SystemTime, UNIX_EPOCH};
use regex::Regex;
use serde::{Deserialize, Serialize};
use crate::engine::{self, eval_json_comparison};
use crate::config::Settings;
use crate::engine::filter_datetime;
use crate::engine::filter_parser::{self, CompareOp, FilterExpr, FilterValue};
use crate::engine::rate::RateWindow;
use crate::config::Settings;
use crate::engine::timestamp::Timestamp;
use crate::engine::{self, eval_json_comparison};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct AlertRule {
@@ -27,7 +27,9 @@ pub struct AlertRule {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum AlertCondition {
Immediate { query: String },
Immediate {
query: String,
},
Threshold {
query: String,
op: CompareOp,
@@ -89,7 +91,12 @@ impl Settings {
enabled: bool,
condition: AlertCondition,
) -> Result<AlertRule, AlertError> {
let rule = AlertRule { id: generate_alert_rule_id(&name), name, enabled, condition };
let rule = AlertRule {
id: generate_alert_rule_id(&name),
name,
enabled,
condition,
};
self.alert_rules.push(rule.clone());
self.save(settings_path)?;
Ok(rule)
@@ -147,36 +154,56 @@ fn backfill_window(
window_ms: i64,
) -> RateWindow {
let since_ms = now.as_millis() - window_ms;
let matching_timestamps =
snapshot.logs_since(before_sequence, since_ms, now.as_millis()).filter_map(
|(sequence, timestamp, text)| {
(eval_alert_query(query, sequence, timestamp, text) == Ok(true)).then_some(timestamp)
},
);
let matching_timestamps = snapshot
.logs_since(before_sequence, since_ms, now.as_millis())
.filter_map(|(sequence, timestamp, text)| {
(eval_alert_query(query, sequence, timestamp, text) == Ok(true)).then_some(timestamp)
});
RateWindow::backfill(matching_timestamps, now, window_ms)
}
fn eval_alert_query(query: &str, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
fn eval_alert_query(
query: &str,
sequence: u64,
timestamp: Timestamp,
text: &str,
) -> Result<bool, String> {
let expr = filter_parser::parse(query).map_err(|err| err.to_string())?;
eval_alert_expr(&expr, sequence, timestamp, text)
}
fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text: &str) -> Result<bool, String> {
fn eval_alert_expr(
expr: &FilterExpr,
sequence: u64,
timestamp: Timestamp,
text: &str,
) -> Result<bool, String> {
match expr {
FilterExpr::Regex(pattern) => {
let re = Regex::new(pattern).map_err(|err| format!("invalid regex in alert query: {err}"))?;
let re = Regex::new(pattern)
.map_err(|err| format!("invalid regex in alert query: {err}"))?;
Ok(re.is_match(text))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_seq" => {
FilterExpr::Comparison {
metric,
param: None,
op,
value,
} if metric == "ottyl_seq" => {
let target = match value {
FilterValue::Number(n) => *n,
FilterValue::Raw(raw) => raw
.parse::<f64>()
.map_err(|_| format!("'ottyl_seq' comparisons require a numeric value, got {raw:?}"))?,
FilterValue::Raw(raw) => raw.parse::<f64>().map_err(|_| {
format!("'ottyl_seq' comparisons require a numeric value, got {raw:?}")
})?,
};
Ok(op.matches((sequence as f64).total_cmp(&target)))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_ts" => {
FilterExpr::Comparison {
metric,
param: None,
op,
value,
} if metric == "ottyl_ts" => {
let target_ms = match value {
FilterValue::Number(n) => *n as i64,
FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, Timestamp::now())
@@ -185,7 +212,12 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
};
Ok(op.matches(timestamp.as_millis().cmp(&target_ms)))
}
FilterExpr::Comparison { metric, param: None, op, value } if metric == "ottyl_fmt" => {
FilterExpr::Comparison {
metric,
param: None,
op,
value,
} if metric == "ottyl_fmt" => {
let target = match value {
FilterValue::Number(n) => n.to_string(),
FilterValue::Raw(raw) => raw.clone(),
@@ -199,12 +231,10 @@ fn eval_alert_expr(expr: &FilterExpr, sequence: u64, timestamp: Timestamp, text:
FilterExpr::JsonComparison { field, op, value } => {
Ok(eval_json_comparison(text, field, *op, value))
}
FilterExpr::And(lhs, rhs) => {
Ok(eval_alert_expr(lhs, sequence, timestamp, text)? && eval_alert_expr(rhs, sequence, timestamp, text)?)
}
FilterExpr::Or(lhs, rhs) => {
Ok(eval_alert_expr(lhs, sequence, timestamp, text)? || eval_alert_expr(rhs, sequence, timestamp, text)?)
}
FilterExpr::And(lhs, rhs) => Ok(eval_alert_expr(lhs, sequence, timestamp, text)?
&& eval_alert_expr(rhs, sequence, timestamp, text)?),
FilterExpr::Or(lhs, rhs) => Ok(eval_alert_expr(lhs, sequence, timestamp, text)?
|| eval_alert_expr(rhs, sequence, timestamp, text)?),
}
}
@@ -222,9 +252,15 @@ pub fn evaluate_alerts(
continue;
}
let state = states.entry(rule.id.clone()).or_default();
if let Some(message) =
evaluate_condition(&rule.condition, state, snapshot, sequence, timestamp, text, &rule.name)
{
if let Some(message) = evaluate_condition(
&rule.condition,
state,
snapshot,
sequence,
timestamp,
text,
&rule.name,
) {
fired.push((rule.clone(), message));
}
}
@@ -241,17 +277,25 @@ fn evaluate_condition(
rule_name: &str,
) -> Option<String> {
match condition {
AlertCondition::Immediate { query } => match eval_alert_query(query, sequence, timestamp, text) {
Ok(true) => Some(format!("{rule_name}: matched line: {text}")),
Ok(false) => None,
Err(err) => {
eprintln!("warning: alert '{rule_name}' query error: {err}");
None
AlertCondition::Immediate { query } => {
match eval_alert_query(query, sequence, timestamp, text) {
Ok(true) => Some(format!("{rule_name}: matched line: {text}")),
Ok(false) => None,
Err(err) => {
eprintln!("warning: alert '{rule_name}' query error: {err}");
None
}
}
},
AlertCondition::Threshold { query, op, threshold, window_ms } => {
}
AlertCondition::Threshold {
query,
op,
threshold,
window_ms,
} => {
if !state.threshold_backfilled {
state.threshold_window = backfill_window(snapshot, query, sequence, timestamp, *window_ms);
state.threshold_window =
backfill_window(snapshot, query, sequence, timestamp, *window_ms);
state.threshold_backfilled = true;
}
match eval_alert_query(query, sequence, timestamp, text) {
@@ -269,7 +313,10 @@ fn evaluate_condition(
}
}
}
AlertCondition::NewValue { field_regex, capture_group } => {
AlertCondition::NewValue {
field_regex,
capture_group,
} => {
let re = match Regex::new(field_regex) {
Ok(re) => re,
Err(err) => {
@@ -284,7 +331,13 @@ fn evaluate_condition(
.insert(value.clone())
.then(|| format!("{rule_name}: new value seen: {value} (line: {text})"))
}
AlertCondition::Ratio { query_a, query_b, op, ratio, window_ms } => {
AlertCondition::Ratio {
query_a,
query_b,
op,
ratio,
window_ms,
} => {
let matched_a = eval_alert_query(query_a, sequence, timestamp, text);
let matched_b = eval_alert_query(query_b, sequence, timestamp, text);
let (matched_a, matched_b) = match (matched_a, matched_b) {
@@ -298,11 +351,13 @@ fn evaluate_condition(
return None;
}
if !state.ratio_a_backfilled {
state.ratio_a_window = backfill_window(snapshot, query_a, sequence, timestamp, *window_ms);
state.ratio_a_window =
backfill_window(snapshot, query_a, sequence, timestamp, *window_ms);
state.ratio_a_backfilled = true;
}
if !state.ratio_b_backfilled {
state.ratio_b_window = backfill_window(snapshot, query_b, sequence, timestamp, *window_ms);
state.ratio_b_window =
backfill_window(snapshot, query_b, sequence, timestamp, *window_ms);
state.ratio_b_backfilled = true;
}
if matched_a {
@@ -326,7 +381,12 @@ fn evaluate_condition(
}
}
pub async fn deliver_for_rule(rule_name: String, message: String, topic: Option<String>, server: Option<String>) {
pub async fn deliver_for_rule(
rule_name: String,
message: String,
topic: Option<String>,
server: Option<String>,
) {
let Some(topic) = topic else {
return;
};
@@ -364,7 +424,9 @@ mod tests {
id: "r1".to_string(),
name: "test rule".to_string(),
enabled: true,
condition: AlertCondition::Immediate { query: query.to_string() },
condition: AlertCondition::Immediate {
query: query.to_string(),
},
}
}
@@ -379,7 +441,9 @@ mod tests {
&settings_path,
"spike".to_string(),
true,
AlertCondition::Immediate { query: r#"regex("error")"#.to_string() },
AlertCondition::Immediate {
query: r#"regex("error")"#.to_string(),
},
)
.unwrap();
assert_eq!(settings.alert_rules.len(), 1);
@@ -393,14 +457,18 @@ mod tests {
&created.id,
"renamed".to_string(),
false,
AlertCondition::Immediate { query: r#"regex("warn")"#.to_string() },
AlertCondition::Immediate {
query: r#"regex("warn")"#.to_string(),
},
)
.unwrap();
assert_eq!(updated.id, created.id);
assert_eq!(updated.name, "renamed");
assert!(!updated.enabled);
settings.remove_alert_rule(&settings_path, &created.id).unwrap();
settings
.remove_alert_rule(&settings_path, &created.id)
.unwrap();
assert!(settings.alert_rules.is_empty());
let reloaded = Settings::load(&settings_path);
assert!(reloaded.alert_rules.is_empty());
@@ -417,7 +485,9 @@ mod tests {
"does-not-exist",
"x".to_string(),
true,
AlertCondition::Immediate { query: r#"regex("x")"#.to_string() },
AlertCondition::Immediate {
query: r#"regex("x")"#.to_string(),
},
);
assert!(matches!(result, Err(AlertError::NotFound)));
@@ -466,12 +536,36 @@ mod tests {
let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
for (seq, ms) in [(0u64, 0i64), (1, 1000)] {
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, seq, Timestamp::from_millis(ms), "error");
assert!(fired.is_empty(), "should not fire before threshold is crossed");
let fired = evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
seq,
Timestamp::from_millis(ms),
"error",
);
assert!(
fired.is_empty(),
"should not fire before threshold is crossed"
);
}
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 2, Timestamp::from_millis(2000), "error");
let fired = evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
2,
Timestamp::from_millis(2000),
"error",
);
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule], &mut states, &index, 3, Timestamp::from_millis(2100), "fine");
let fired = evaluate_alerts(
&[rule],
&mut states,
&index,
3,
Timestamp::from_millis(2100),
"fine",
);
assert!(fired.is_empty());
}
@@ -490,8 +584,28 @@ mod tests {
};
let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
assert!(evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "error").is_empty());
assert!(evaluate_alerts(&[rule], &mut states, &index, 1, Timestamp::from_millis(5_000), "error").is_empty());
assert!(
evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
0,
Timestamp::from_millis(0),
"error"
)
.is_empty()
);
assert!(
evaluate_alerts(
&[rule],
&mut states,
&index,
1,
Timestamp::from_millis(5_000),
"error"
)
.is_empty()
);
}
#[test]
@@ -507,11 +621,32 @@ mod tests {
};
let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "agent=curl");
let fired = evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
0,
Timestamp::from_millis(0),
"agent=curl",
);
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 1, Timestamp::from_millis(1), "agent=curl");
let fired = evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
1,
Timestamp::from_millis(1),
"agent=curl",
);
assert!(fired.is_empty());
let fired = evaluate_alerts(&[rule], &mut states, &index, 2, Timestamp::from_millis(2), "agent=wget");
let fired = evaluate_alerts(
&[rule],
&mut states,
&index,
2,
Timestamp::from_millis(2),
"agent=wget",
);
assert_eq!(fired.len(), 1);
}
@@ -531,12 +666,43 @@ mod tests {
};
let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
assert!(evaluate_alerts(&[rule.clone()], &mut states, &index, 0, Timestamp::from_millis(0), "error").is_empty());
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 1, Timestamp::from_millis(1), "request");
assert!(
evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
0,
Timestamp::from_millis(0),
"error"
)
.is_empty()
);
let fired = evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
1,
Timestamp::from_millis(1),
"request",
);
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule.clone()], &mut states, &index, 2, Timestamp::from_millis(2), "request");
let fired = evaluate_alerts(
std::slice::from_ref(&rule),
&mut states,
&index,
2,
Timestamp::from_millis(2),
"request",
);
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule], &mut states, &index, 3, Timestamp::from_millis(3), "request");
let fired = evaluate_alerts(
&[rule],
&mut states,
&index,
3,
Timestamp::from_millis(3),
"request",
);
assert!(fired.is_empty());
}
@@ -547,7 +713,7 @@ mod tests {
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(
&[rule.clone()],
std::slice::from_ref(&rule),
&mut states,
&index,
0,
@@ -556,7 +722,14 @@ mod tests {
);
assert_eq!(fired.len(), 1);
let fired = evaluate_alerts(&[rule], &mut states, &index, 1, Timestamp::from_millis(1), "plain text");
let fired = evaluate_alerts(
&[rule],
&mut states,
&index,
1,
Timestamp::from_millis(1),
"plain text",
);
assert!(fired.is_empty());
}
@@ -565,15 +738,22 @@ mod tests {
let rule = immediate_rule("some_metric[param]==5");
let mut states = HashMap::new();
let index = engine::EngineSnapshot::default();
let fired = evaluate_alerts(&[rule], &mut states, &index, 0, Timestamp::from_millis(0), "anything");
let fired = evaluate_alerts(
&[rule],
&mut states,
&index,
0,
Timestamp::from_millis(0),
"anything",
);
assert!(fired.is_empty());
}
#[tokio::test]
async fn deliver_ntfy_posts_message_to_server_topic_path() {
use axum::Router;
use axum::extract::State;
use axum::routing::post;
use axum::Router;
use std::sync::{Arc, Mutex};
use tokio::net::TcpListener;
@@ -590,7 +770,9 @@ mod tests {
}
let captured = Captured::default();
let app = Router::new().route("/{topic}", post(capture)).with_state(captured.clone());
let app = Router::new()
.route("/{topic}", post(capture))
.with_state(captured.clone());
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
@@ -598,7 +780,9 @@ mod tests {
});
let server = format!("http://{addr}");
deliver_ntfy(&server, "alerts", "hello world").await.unwrap();
deliver_ntfy(&server, "alerts", "hello world")
.await
.unwrap();
let (topic, body) = captured.0.lock().unwrap().clone().unwrap();
assert_eq!(topic, "alerts");
src/engine/data_value.rs
+10 -3
View File
@@ -39,7 +39,8 @@ pub fn extract_json_field(text: &str, field_path: &str) -> Option<String> {
}
pub fn flatten_json_object(text: &str) -> Vec<(String, String)> {
let Ok(serde_json::Value::Object(root)) = serde_json::from_str::<serde_json::Value>(text) else {
let Ok(serde_json::Value::Object(root)) = serde_json::from_str::<serde_json::Value>(text)
else {
return Vec::new();
};
let mut results = Vec::new();
@@ -53,7 +54,11 @@ fn flatten_json_object_into(
results: &mut Vec<(String, String)>,
) {
for (key, value) in object {
let path = if prefix.is_empty() { key.clone() } else { format!("{prefix}.{key}") };
let path = if prefix.is_empty() {
key.clone()
} else {
format!("{prefix}.{key}")
};
match value {
serde_json::Value::String(s) => results.push((path, s.clone())),
serde_json::Value::Number(n) => results.push((path, n.to_string())),
@@ -78,7 +83,9 @@ pub enum MetricPrimitiveKind {
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize)]
pub enum DataValue {
LogLiteral { text: String },
LogLiteral {
text: String,
},
ParamDeclaration(ParamDeclaration),
MetricDeclaration {
message_match_regex: String,
src/engine/filter_datetime.rs
+10 -11
View File
@@ -58,10 +58,7 @@ fn try_relative(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTim
},
};
let looks_like_duration = rest
.chars()
.next()
.is_some_and(|c| c.is_ascii_digit())
let looks_like_duration = rest.chars().next().is_some_and(|c| c.is_ascii_digit())
&& rest.chars().any(|c| c.is_ascii_alphabetic())
&& !rest.contains('-')
&& !rest.contains(':');
@@ -107,10 +104,7 @@ fn try_full_datetime(input: &str) -> Option<Result<Timestamp, DateTimeParseError
}
}
fn try_time_of_day(
input: &str,
now: Timestamp,
) -> Option<Result<Timestamp, DateTimeParseError>> {
fn try_time_of_day(input: &str, now: Timestamp) -> Option<Result<Timestamp, DateTimeParseError>> {
if !input.contains(':') {
return None;
}
@@ -129,9 +123,14 @@ fn try_time_of_day(
};
let now_millis = now.as_millis();
let now_dt = Utc.timestamp_millis_opt(now_millis).single().unwrap_or_else(|| {
Utc.timestamp_millis_opt(0).single().expect("epoch is valid")
});
let now_dt = Utc
.timestamp_millis_opt(now_millis)
.single()
.unwrap_or_else(|| {
Utc.timestamp_millis_opt(0)
.single()
.expect("epoch is valid")
});
let today = now_dt.date_naive();
let dt = Utc.from_utc_datetime(&today.and_time(time));
Some(Ok(timestamp_from_datetime(dt)))
src/engine/filter_parser.rs
+9 -13
View File
@@ -88,9 +88,7 @@ pub fn parse(input: &str) -> Result<FilterExpr, FilterParseError> {
fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
debug_assert_eq!(pair.as_rule(), Rule::or_expr);
let mut and_exprs = pair
.into_inner()
.filter(|p| p.as_rule() != Rule::or_kw);
let mut and_exprs = pair.into_inner().filter(|p| p.as_rule() != Rule::or_kw);
let first = and_exprs
.next()
.ok_or_else(|| FilterParseError::new("malformed 'or' expression"))?;
@@ -104,9 +102,7 @@ fn build_or_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
fn build_and_expr(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
debug_assert_eq!(pair.as_rule(), Rule::and_expr);
let mut terms = pair
.into_inner()
.filter(|p| p.as_rule() != Rule::and_kw);
let mut terms = pair.into_inner().filter(|p| p.as_rule() != Rule::and_kw);
let first = terms
.next()
.ok_or_else(|| FilterParseError::new("malformed 'and' expression"))?;
@@ -182,9 +178,9 @@ fn build_comparison(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
fn build_json_comparison(pair: Pair<Rule>) -> Result<FilterExpr, FilterParseError> {
let mut inner = pair.into_inner();
let field_pair = inner
.next()
.ok_or_else(|| FilterParseError::new("malformed json[...] comparison: missing field name"))?;
let field_pair = inner.next().ok_or_else(|| {
FilterParseError::new("malformed json[...] comparison: missing field name")
})?;
let field = field_pair.as_str().to_string();
let op_pair = inner
@@ -355,8 +351,8 @@ mod tests {
#[test]
fn parses_and_or_with_and_binding_tighter() {
let expr = parse(r#"regex("error") and ottyl_seq<5 or ottyl_ts>12:00"#)
.expect("should parse");
let expr =
parse(r#"regex("error") and ottyl_seq<5 or ottyl_ts>12:00"#).expect("should parse");
let expected = FilterExpr::Or(
Box::new(FilterExpr::And(
@@ -381,8 +377,8 @@ mod tests {
#[test]
fn parenthesized_expression_overrides_precedence() {
let expr = parse(r#"regex("error") and (ottyl_seq<5 or ottyl_ts>12:00)"#)
.expect("should parse");
let expr =
parse(r#"regex("error") and (ottyl_seq<5 or ottyl_ts>12:00)"#).expect("should parse");
let expected = FilterExpr::And(
Box::new(FilterExpr::Regex("error".to_string())),
src/engine/handle.rs
+5 -1
View File
@@ -110,7 +110,11 @@ fn new_channel_and_handle(
let reader = IndexReader {
factory: read.factory(),
};
(EngineHandle { tx, reader }, rx, IndexWriter { handle: write })
(
EngineHandle { tx, reader },
rx,
IndexWriter { handle: write },
)
}
pub fn spawn(settings: SettingsHandle, channel_capacity: usize) -> EngineHandle {
src/engine/ingest.rs
+40 -24
View File
@@ -2,8 +2,8 @@ use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use crate::consts::{METRICS_POLL_INTERVAL, STDIN_POLL_INTERVAL as POLL_INTERVAL};
use crate::engine::{EngineHandle, EngineMessage, IngestedLine, MetricSample};
use crate::engine::timestamp::Timestamp;
use crate::engine::{EngineHandle, EngineMessage, IngestedLine, MetricSample};
use tokio::fs::File;
use tokio::io::{AsyncBufReadExt, BufReader};
@@ -67,15 +67,12 @@ pub async fn run_file(engine: EngineHandle, path: PathBuf) {
let mut chunk = String::new();
match reader.read_line(&mut chunk).await {
Ok(0) => {
match tokio::fs::metadata(&path).await {
Ok(meta) => {
let new_id = file_id(&meta);
if new_id != id || meta.len() < offset {
continue 'reopen;
}
id = new_id;
if let Ok(meta) = tokio::fs::metadata(&path).await {
let new_id = file_id(&meta);
if new_id != id || meta.len() < offset {
continue 'reopen;
}
Err(_) => {}
id = new_id;
}
tokio::time::sleep(POLL_INTERVAL).await;
}
@@ -83,9 +80,7 @@ pub async fn run_file(engine: EngineHandle, path: PathBuf) {
offset += n as u64;
pending.push_str(&chunk);
if pending.ends_with('\n') {
let text = pending
.trim_end_matches(['\n', '\r'])
.to_string();
let text = pending.trim_end_matches(['\n', '\r']).to_string();
pending.clear();
let line = IngestedLine {
@@ -128,7 +123,12 @@ pub async fn run_metrics_poll(engine: EngineHandle, url: String) {
Ok(body) => {
let now = Timestamp::now();
for sample in parse_openmetrics(&body, now) {
if engine.tx.send(EngineMessage::IngestMetricSample(sample)).await.is_err() {
if engine
.tx
.send(EngineMessage::IngestMetricSample(sample))
.await
.is_err()
{
return;
}
}
@@ -140,7 +140,9 @@ pub async fn run_metrics_poll(engine: EngineHandle, url: String) {
}
fn parse_openmetrics(body: &str, now: Timestamp) -> Vec<MetricSample> {
body.lines().filter_map(|line| parse_openmetrics_line(line, now)).collect()
body.lines()
.filter_map(|line| parse_openmetrics_line(line, now))
.collect()
}
fn parse_openmetrics_line(line: &str, now: Timestamp) -> Option<MetricSample> {
@@ -161,7 +163,10 @@ fn parse_openmetrics_line(line: &str, now: Timestamp) -> Option<MetricSample> {
let (metric_name, labels) = match name_and_labels.find('{') {
Some(open) => {
let close = name_and_labels.rfind('}')?;
(name_and_labels[..open].to_string(), parse_openmetrics_labels(&name_and_labels[open + 1..close]))
(
name_and_labels[..open].to_string(),
parse_openmetrics_labels(&name_and_labels[open + 1..close]),
)
}
None => (name_and_labels.to_string(), BTreeMap::new()),
};
@@ -169,13 +174,23 @@ fn parse_openmetrics_line(line: &str, now: Timestamp) -> Option<MetricSample> {
return None;
}
Some(MetricSample { metric_name, labels, value, timestamp })
Some(MetricSample {
metric_name,
labels,
value,
timestamp,
})
}
fn parse_openmetrics_labels(raw: &str) -> BTreeMap<String, String> {
raw.split(',')
.filter_map(|pair| pair.split_once('='))
.map(|(name, value)| (name.trim().to_string(), value.trim().trim_matches('"').to_string()))
.map(|(name, value)| {
(
name.trim().to_string(),
value.trim().trim_matches('"').to_string(),
)
})
.collect()
}
@@ -197,10 +212,8 @@ mod openmetrics_tests {
#[test]
fn parses_labels_and_explicit_timestamp() {
let now = Timestamp::from_millis(1000);
let samples = parse_openmetrics(
r#"http_requests_total{method="GET",path="/a"} 7 5000"#,
now,
);
let samples =
parse_openmetrics(r#"http_requests_total{method="GET",path="/a"} 7 5000"#, now);
assert_eq!(samples.len(), 1);
let sample = &samples[0];
assert_eq!(sample.metric_name, "http_requests_total");
@@ -223,9 +236,9 @@ mod openmetrics_tests {
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
use std::io::Write;
use crate::engine::test_handle;
use std::io::Write;
use std::time::Duration;
use tokio::sync::mpsc;
async fn recv_text(rx: &mut mpsc::Receiver<EngineMessage>) -> String {
@@ -250,7 +263,10 @@ mod tests {
assert_eq!(recv_text(&mut rx).await, "first");
let mut f = std::fs::OpenOptions::new().append(true).open(&path).unwrap();
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap();
writeln!(f, "second").unwrap();
assert_eq!(recv_text(&mut rx).await, "second");
}
src/engine/logfmt.rs
+81 -21
View File
@@ -1,9 +1,9 @@
use nom::IResult;
use nom::branch::alt;
use nom::bytes::complete::{escaped, tag, take_until, take_while1};
use nom::character::complete::{char, none_of, one_of};
use nom::error::{Error, ErrorKind};
use nom::sequence::delimited;
use nom::IResult;
use regex::Regex;
use std::sync::LazyLock;
@@ -16,7 +16,10 @@ fn key(input: &str) -> IResult<&str, &str> {
fn quoted_value(input: &str) -> IResult<&str, &str> {
delimited(
char('"'),
alt((escaped(none_of("\\\""), '\\', one_of("\"\\nt")), nom::combinator::success(""))),
alt((
escaped(none_of("\\\""), '\\', one_of("\"\\nt")),
nom::combinator::success(""),
)),
char('"'),
)(input)
}
@@ -85,7 +88,8 @@ fn is_separator_run(run: &str) -> bool {
run.len() >= 2 || run.contains('\t')
}
static WHITESPACE_RUN: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"[ \t]+").expect("static regex must compile"));
static WHITESPACE_RUN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"[ \t]+").expect("static regex must compile"));
fn split_columns_with_offsets(line: &str, max_columns: Option<usize>) -> Vec<(usize, &str)> {
let mut columns = Vec::new();
@@ -107,7 +111,10 @@ fn split_columns_with_offsets(line: &str, max_columns: Option<usize>) -> Vec<(us
}
fn split_columns(line: &str, max_columns: Option<usize>) -> Vec<&str> {
split_columns_with_offsets(line, max_columns).into_iter().map(|(_, s)| s).collect()
split_columns_with_offsets(line, max_columns)
.into_iter()
.map(|(_, s)| s)
.collect()
}
static COLUMN_NAME: LazyLock<Regex> =
@@ -151,13 +158,21 @@ impl LogStreamHeader {
}
let leading_ws = slice.len() - slice.trim_start().len();
let start = line[..byte_start + leading_ws].chars().count();
columns.push(LogStreamColumn { name: trimmed.to_string(), start });
columns.push(LogStreamColumn {
name: trimmed.to_string(),
start,
});
}
if columns.len() < 2 || !columns.iter().all(|c| COLUMN_NAME.is_match(&c.name)) {
return None;
}
let tab_delimited = WHITESPACE_RUN.find_iter(line).any(|m| m.as_str().contains('\t'));
Some(LogStreamHeader { columns, tab_delimited })
let tab_delimited = WHITESPACE_RUN
.find_iter(line)
.any(|m| m.as_str().contains('\t'));
Some(LogStreamHeader {
columns,
tab_delimited,
})
}
pub fn split_row<'a>(&self, line: &'a str) -> Vec<&'a str> {
@@ -210,7 +225,9 @@ impl LogStreamHeader {
return pattern;
}
let start = self.columns.last().map(|c| c.start).unwrap_or(0);
format!("^.{{{start}}}{COLUMN_VALUE_PATTERN}{COLUMN_SEPARATOR_PATTERN}(?P<{capture_group}>.*)$")
format!(
"^.{{{start}}}{COLUMN_VALUE_PATTERN}{COLUMN_SEPARATOR_PATTERN}(?P<{capture_group}>.*)$"
)
}
}
@@ -240,7 +257,9 @@ fn py_log_prefix(input: &str) -> IResult<&str, (&str, &str, &str, &str)> {
let (input, level) = take_while1(|c: char| c.is_ascii_uppercase())(input)?;
let (input, _) = char(' ')(input)?;
let (input, logger) =
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')(input)?;
take_while1(|c: char| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')(
input,
)?;
let (input, _) = char(' ')(input)?;
Ok((input, (date, time, level, logger)))
}
@@ -323,7 +342,10 @@ mod tests {
#[test]
fn detects_key_value_lines_as_logfmt() {
assert_eq!(detect_format(r#"ts=1788648105773 msg="entity_moved" x1=-4"#), "logfmt");
assert_eq!(
detect_format(r#"ts=1788648105773 msg="entity_moved" x1=-4"#),
"logfmt"
);
}
#[test]
@@ -371,7 +393,10 @@ mod tests {
fn detects_log_stream_header_columns_generically() {
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).expect("fixture header must be detected");
assert_eq!(column_names(&header), vec!["Timestamp", "Thread", "Type", "Activity", "PID", "TTL"]);
assert_eq!(
column_names(&header),
vec!["Timestamp", "Thread", "Type", "Activity", "PID", "TTL"]
);
assert!(!header.tab_delimited);
}
@@ -400,7 +425,10 @@ mod tests {
assert_eq!(row[3], "0x0");
assert_eq!(row[4], "606");
assert_eq!(row[5], "0");
assert_eq!(row[6], "WindowServer: (HSTouchHIDService) something happened");
assert_eq!(
row[6],
"WindowServer: (HSTouchHIDService) something happened"
);
}
#[test]
@@ -426,32 +454,57 @@ mod tests {
let line = "2026-09-06 01:48:54.518117+0100 0x4dfed2 Default 0x0 606 0 WindowServer: (HSTouchHIDService) something happened";
let type_re = Regex::new(&header.column_pattern(2, "Type")).unwrap();
assert_eq!(type_re.captures(line).unwrap().name("Type").unwrap().as_str(), "Default");
assert_eq!(
type_re
.captures(line)
.unwrap()
.name("Type")
.unwrap()
.as_str(),
"Default"
);
let pid_re = Regex::new(&header.column_pattern(4, "PID")).unwrap();
assert_eq!(pid_re.captures(line).unwrap().name("PID").unwrap().as_str(), "606");
assert_eq!(
pid_re.captures(line).unwrap().name("PID").unwrap().as_str(),
"606"
);
let message_re = Regex::new(&header.message_pattern("message")).unwrap();
assert_eq!(
message_re.captures(line).unwrap().name("message").unwrap().as_str(),
message_re
.captures(line)
.unwrap()
.name("message")
.unwrap()
.as_str(),
"WindowServer: (HSTouchHIDService) something happened"
);
let timestamp_re = Regex::new(&header.column_pattern(0, "Timestamp")).unwrap();
assert_eq!(
timestamp_re.captures(line).unwrap().name("Timestamp").unwrap().as_str(),
timestamp_re
.captures(line)
.unwrap()
.name("Timestamp")
.unwrap()
.as_str(),
"2026-09-06 01:48:54.518117+0100"
);
}
#[test]
fn is_log_stream_line_rejects_non_columnar_lines_that_a_fixed_offset_column_regex_would_still_match() {
fn is_log_stream_line_rejects_non_columnar_lines_that_a_fixed_offset_column_regex_would_still_match()
{
let header_line = LOG_STREAM_FIXTURE.lines().next().unwrap();
let header = LogStreamHeader::detect(header_line).unwrap();
let timestamp_re = Regex::new(&header.column_pattern(0, "Timestamp")).unwrap();
let continuation_line = "so_gencnt: 4042707 so_state: 0x0";
assert!(timestamp_re.captures(continuation_line).is_some(), "sanity: regex alone still matches garbage");
assert!(
timestamp_re.captures(continuation_line).is_some(),
"sanity: regex alone still matches garbage"
);
assert!(!is_log_stream_line(continuation_line));
assert!(!is_log_stream_line(header_line));
assert!(!is_log_stream_line("}"));
@@ -473,7 +526,10 @@ mod tests {
#[test]
fn detects_go_log_shaped_lines() {
for line in GO_LOG_FIXTURE.lines() {
assert!(is_go_log_line(line), "expected go-log line to match: {line}");
assert!(
is_go_log_line(line),
"expected go-log line to match: {line}"
);
assert_eq!(detect_format(line), "go-log");
}
}
@@ -481,7 +537,10 @@ mod tests {
#[test]
fn detects_py_log_shaped_lines() {
for line in PY_LOG_FIXTURE.lines() {
assert!(is_py_log_line(line), "expected py-log line to match: {line}");
assert!(
is_py_log_line(line),
"expected py-log line to match: {line}"
);
assert_eq!(detect_format(line), "py-log");
}
}
@@ -514,7 +573,8 @@ mod tests {
#[test]
fn py_log_prefix_extracts_date_time_level_and_logger() {
let (rest, (date, time, level, logger)) =
py_log_prefix("2026-09-06 10:35:55,217 INFO delete-py-log event=eventC labelC=1").unwrap();
py_log_prefix("2026-09-06 10:35:55,217 INFO delete-py-log event=eventC labelC=1")
.unwrap();
assert_eq!(date, "2026-09-06");
assert_eq!(time, "10:35:55,217");
assert_eq!(level, "INFO");
src/engine/message.rs
+8 -2
View File
@@ -4,7 +4,9 @@ use crate::api::dto::{
BucketValueDto, LogEntryDto, LogsParams, MetricDeclarationDto, ParamDeclarationDto,
PromqlSeriesDto, StatsDto, TimelineBucketDto, TimelineParams,
};
use crate::engine::{ClearParamsResult, DeclareMetricRequest, DeclareParamRequest, IngestedLine, MetricSample};
use crate::engine::{
ClearParamsResult, DeclareMetricRequest, DeclareParamRequest, IngestedLine, MetricSample,
};
use std::collections::BTreeMap;
use tokio::sync::oneshot;
@@ -23,7 +25,11 @@ pub enum EngineQuery {
since_sequence: Option<u64>,
limit: Option<usize>,
},
Promql { query: String, start_ms: Option<i64>, end_ms: Option<i64> },
Promql {
query: String,
start_ms: Option<i64>,
end_ms: Option<i64>,
},
}
#[allow(dead_code)]
src/engine/mod.rs
+2 -2
View File
@@ -16,11 +16,11 @@ pub mod timestamp;
#[cfg(test)]
mod tests;
pub use handle::{spawn, EngineHandle};
#[cfg(test)]
pub(crate) use handle::test_handle;
pub use handle::{EngineHandle, spawn};
pub use message::{EngineMessage, EngineQuery, EngineResponse};
pub(crate) use snapshot::{eval_json_comparison, EngineSnapshot};
pub(crate) use snapshot::{EngineSnapshot, eval_json_comparison};
use crate::engine::data_value::ParamDeclaration;
use crate::engine::timestamp::Timestamp;
src/engine/promql.rs
+46 -11
View File
@@ -38,7 +38,10 @@ impl PromqlQuery {
let query = parser.parse_query()?;
parser.skip_whitespace();
if !parser.is_at_end() {
return Err(format!("unexpected trailing input: '{}'", parser.remaining()));
return Err(format!(
"unexpected trailing input: '{}'",
parser.remaining()
));
}
if query.agg == Some(AggOp::Rate) && query.range.is_none() {
return Err("rate() requires a range selector, e.g. rate(my_metric[5m])".to_string());
@@ -113,7 +116,9 @@ impl<'a> Parser<'a> {
fn parse_quoted_string(&mut self) -> Result<String, String> {
self.expect_char('"')?;
let rest = self.remaining();
let end = rest.find('"').ok_or("unterminated string literal (missing closing '\"')")?;
let end = rest
.find('"')
.ok_or("unterminated string literal (missing closing '\"')")?;
let value = &rest[..end];
if value.contains('\\') {
return Err("escape sequences in label values are not supported".to_string());
@@ -185,7 +190,9 @@ impl<'a> Parser<'a> {
return Ok(Some(RangeSelector::BareWidth(digits as f64)));
}
let unit = self.peek().ok_or("range selector is missing a duration unit or ']'")?;
let unit = self
.peek()
.ok_or("range selector is missing a duration unit or ']'")?;
let seconds_per_unit: i64 = match unit {
's' => 1,
'm' => 60,
@@ -272,7 +279,11 @@ impl<'a> Parser<'a> {
_ => None,
};
self.skip_whitespace();
let prefix_by_labels = if agg.is_some() { self.parse_by_clause()? } else { Vec::new() };
let prefix_by_labels = if agg.is_some() {
self.parse_by_clause()?
} else {
Vec::new()
};
self.skip_whitespace();
if agg.is_some() && self.peek() == Some('(') {
self.expect_char('(')?;
@@ -314,8 +325,11 @@ impl<'a> Parser<'a> {
.to_string(),
);
}
inner.by_labels =
if prefix_by_labels.is_empty() { trailing_by_labels } else { prefix_by_labels };
inner.by_labels = if prefix_by_labels.is_empty() {
trailing_by_labels
} else {
prefix_by_labels
};
Ok(inner)
} else {
self.pos = checkpoint;
@@ -343,7 +357,10 @@ mod tests {
assert_eq!(query.metric_name, "requests");
assert_eq!(
query.matchers,
vec![LabelMatcher { name: "path".to_string(), value: "/a".to_string() }]
vec![LabelMatcher {
name: "path".to_string(),
value: "/a".to_string()
}]
);
}
@@ -472,7 +489,13 @@ mod tests {
fn parses_gauge_with_no_range_selector() {
let query = PromqlQuery::parse(r#"gauge(connections{host="a"})"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Gauge));
assert_eq!(query.matchers, vec![LabelMatcher { name: "host".to_string(), value: "a".to_string() }]);
assert_eq!(
query.matchers,
vec![LabelMatcher {
name: "host".to_string(),
value: "a".to_string()
}]
);
}
#[test]
@@ -497,7 +520,13 @@ mod tests {
let query = PromqlQuery::parse(r#"avg(rate(requests{path="/a"}[5m]))"#).unwrap();
assert_eq!(query.agg, Some(AggOp::Avg));
assert!(query.inner_rate);
assert_eq!(query.matchers, vec![LabelMatcher { name: "path".to_string(), value: "/a".to_string() }]);
assert_eq!(
query.matchers,
vec![LabelMatcher {
name: "path".to_string(),
value: "/a".to_string()
}]
);
}
#[test]
@@ -518,7 +547,10 @@ mod tests {
fn parses_sum_of_rate_with_by_clause_multiple_labels() {
let query = PromqlQuery::parse("sum(rate(requests[1m])) by (path, method)").unwrap();
assert!(query.inner_rate);
assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]);
assert_eq!(
query.by_labels,
vec!["path".to_string(), "method".to_string()]
);
}
#[test]
@@ -548,7 +580,10 @@ mod tests {
#[test]
fn parses_sum_with_prefix_by_clause_multiple_labels() {
let query = PromqlQuery::parse("sum by (path, method) (requests)").unwrap();
assert_eq!(query.by_labels, vec!["path".to_string(), "method".to_string()]);
assert_eq!(
query.by_labels,
vec!["path".to_string(), "method".to_string()]
);
}
#[test]
src/engine/rate.rs
+17 -5
View File
@@ -20,7 +20,10 @@ pub fn rate_at(points: &[(Timestamp, f64)], now: Timestamp, window_ms: i64) -> f
}
pub fn windowed_rate_series(points: &[(Timestamp, f64)], window_ms: i64) -> Vec<(Timestamp, f64)> {
points.iter().map(|(ts, _)| (*ts, rate_at(points, *ts, window_ms))).collect()
points
.iter()
.map(|(ts, _)| (*ts, rate_at(points, *ts, window_ms)))
.collect()
}
#[derive(Debug, Default, Clone)]
@@ -29,7 +32,11 @@ pub struct RateWindow {
}
impl RateWindow {
pub fn backfill(timestamps: impl IntoIterator<Item = Timestamp>, now: Timestamp, window_ms: i64) -> Self {
pub fn backfill(
timestamps: impl IntoIterator<Item = Timestamp>,
now: Timestamp,
window_ms: i64,
) -> Self {
let mut window = Self::default();
for ts in timestamps {
window.push(ts, window_ms);
@@ -94,9 +101,14 @@ mod tests {
for ms in millis {
window.push(Timestamp::from_millis(ms), 1000);
}
let points: Vec<(Timestamp, f64)> =
millis.iter().map(|ms| (Timestamp::from_millis(*ms), 1.0)).collect();
assert_eq!(window.count(), count_in_window(&points, Timestamp::from_millis(1200), 1000));
let points: Vec<(Timestamp, f64)> = millis
.iter()
.map(|ms| (Timestamp::from_millis(*ms), 1.0))
.collect();
assert_eq!(
window.count(),
count_in_window(&points, Timestamp::from_millis(1200), 1000)
);
}
#[test]
src/engine/snapshot/filter.rs
+45 -11
View File
@@ -12,11 +12,24 @@ use std::collections::HashSet;
pub(super) enum CompiledFilter {
Regex(Regex),
Sequence { op: CompareOp, target: f64 },
Timestamp { op: CompareOp, target_ms: i64 },
Fmt { op: CompareOp, target: String },
Sequence {
op: CompareOp,
target: f64,
},
Timestamp {
op: CompareOp,
target_ms: i64,
},
Fmt {
op: CompareOp,
target: String,
},
MetricMatch(HashSet<u64>),
Json { field: String, op: CompareOp, value: FilterValue },
Json {
field: String,
op: CompareOp,
value: FilterValue,
},
And(Box<CompiledFilter>, Box<CompiledFilter>),
Or(Box<CompiledFilter>, Box<CompiledFilter>),
}
@@ -32,7 +45,11 @@ impl CompiledFilter {
Ok(CompiledFilter::Sequence { op, target })
}
pub(super) fn from_ts_comparison(op: CompareOp, value: &FilterValue, now: Timestamp) -> Result<Self, String> {
pub(super) fn from_ts_comparison(
op: CompareOp,
value: &FilterValue,
now: Timestamp,
) -> Result<Self, String> {
let target_ms = match value {
FilterValue::Number(n) => *n as i64,
FilterValue::Raw(raw) => filter_datetime::parse_datetime(raw, now)
@@ -52,7 +69,12 @@ impl CompiledFilter {
}
impl MetricPrimitive {
pub(super) fn compare(&self, op: CompareOp, value: &FilterValue, now: Timestamp) -> Result<bool, String> {
pub(super) fn compare(
&self,
op: CompareOp,
value: &FilterValue,
now: Timestamp,
) -> Result<bool, String> {
let ordering = match (self, value) {
(MetricPrimitive::Number(n), FilterValue::Number(v)) => n.as_f64().partial_cmp(v),
(MetricPrimitive::Number(n), FilterValue::Raw(raw)) => {
@@ -65,10 +87,13 @@ impl MetricPrimitive {
ts.as_millis().partial_cmp(&(*v as i64))
}
(MetricPrimitive::Timestamp(ts), FilterValue::Raw(raw)) => {
let target = filter_datetime::parse_datetime(raw, now).map_err(|err| err.to_string())?;
let target =
filter_datetime::parse_datetime(raw, now).map_err(|err| err.to_string())?;
ts.as_millis().partial_cmp(&target.as_millis())
}
(MetricPrimitive::String(s), FilterValue::Raw(raw)) => s.as_str().partial_cmp(raw.as_str()),
(MetricPrimitive::String(s), FilterValue::Raw(raw)) => {
s.as_str().partial_cmp(raw.as_str())
}
(MetricPrimitive::String(s), FilterValue::Number(n)) => {
s.as_str().partial_cmp(n.to_string().as_str())
}
@@ -91,7 +116,12 @@ impl MetricPrimitive {
}
}
pub(crate) fn eval_json_comparison(text: &str, field: &str, op: CompareOp, value: &FilterValue) -> bool {
pub(crate) fn eval_json_comparison(
text: &str,
field: &str,
op: CompareOp,
value: &FilterValue,
) -> bool {
let Ok(parsed) = serde_json::from_str::<serde_json::Value>(text) else {
return false;
};
@@ -136,7 +166,9 @@ fn compare_json_value(field_value: &serde_json::Value, op: CompareOp, value: &Fi
Err(_) => false,
}
}
serde_json::Value::Null | serde_json::Value::Array(_) | serde_json::Value::Object(_) => false,
serde_json::Value::Null | serde_json::Value::Array(_) | serde_json::Value::Object(_) => {
false
}
}
}
@@ -167,7 +199,9 @@ impl CompiledFilter {
op.matches(crate::engine::logfmt::detect_format(text).cmp(target.as_str()))
}
CompiledFilter::MetricMatch(sequences) => sequences.contains(&sequence),
CompiledFilter::Json { field, op, value } => eval_json_comparison(text, field, *op, value),
CompiledFilter::Json { field, op, value } => {
eval_json_comparison(text, field, *op, value)
}
CompiledFilter::And(lhs, rhs) => {
lhs.eval(sequence, timestamp, text) && rhs.eval(sequence, timestamp, text)
}
src/engine/snapshot/metrics.rs
+4 -2
View File
@@ -6,7 +6,10 @@ use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::metric_primitive::{MetricPrimitive, Number};
use crate::engine::timestamp::Timestamp;
pub(super) fn parse_metric_primitive(raw: &str, kind: MetricPrimitiveKind) -> Option<MetricPrimitive> {
pub(super) fn parse_metric_primitive(
raw: &str,
kind: MetricPrimitiveKind,
) -> Option<MetricPrimitive> {
match kind {
MetricPrimitiveKind::String => Some(MetricPrimitive::String(raw.to_string())),
MetricPrimitiveKind::Number => raw
@@ -24,4 +27,3 @@ pub(super) fn parse_metric_primitive(raw: &str, kind: MetricPrimitiveKind) -> Op
MetricPrimitiveKind::Boolean => raw.parse::<bool>().ok().map(MetricPrimitive::Boolean),
}
}
src/engine/snapshot/mod.rs
+413 -199
View File
File diff suppressed because it is too large Load Diff
src/engine/snapshot/params.rs
+20 -7
View File
@@ -1,23 +1,34 @@
//! Param declaration: validating and storing explicit declarations, and the auto-detection
//! logic that derives declarations from logfmt/json/log-stream-shaped lines as they're ingested.
#[cfg(test)]
use crate::engine::DeclareParamRequest;
use crate::engine::data_value::MetricPrimitiveKind;
#[cfg(test)]
use crate::engine::snapshot::EngineSnapshot;
#[cfg(test)]
use crate::engine::DeclareParamRequest;
use regex::Regex;
#[cfg(test)]
pub(crate) fn declare_param(snapshot: &mut EngineSnapshot, request: DeclareParamRequest) -> Result<(), String> {
pub(crate) fn declare_param(
snapshot: &mut EngineSnapshot,
request: DeclareParamRequest,
) -> Result<(), String> {
snapshot.validate_param_declaration(&request)?;
snapshot.insert_param_declaration(request);
Ok(())
}
pub(crate) fn sanitize_group_name(key: &str) -> String {
let mut group: String =
key.chars().map(|c| if c.is_ascii_alphanumeric() || c == '_' { c } else { '_' }).collect();
let mut group: String = key
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '_' {
c
} else {
'_'
}
})
.collect();
if group.chars().next().is_none_or(|c| c.is_ascii_digit()) {
group.insert(0, '_');
}
@@ -41,7 +52,6 @@ pub(crate) fn build_auto_param_pattern(key: &str, group: &str, quoted: bool) ->
}
}
pub(crate) struct AutoParamCandidate {
pub(crate) name: String,
pub(crate) value: String,
@@ -94,6 +104,9 @@ pub(crate) fn extract_param_raw(
}
match json_field_path.filter(|path| !path.is_empty()) {
Some(field_path) => crate::engine::data_value::extract_json_field(text, field_path),
None => regex.captures(text)?.name(capture_group).map(|m| m.as_str().to_string()),
None => regex
.captures(text)?
.name(capture_group)
.map(|m| m.as_str().to_string()),
}
}
src/engine/snapshot/query.rs
+121 -59
View File
@@ -2,8 +2,8 @@
//! stats, declaration listing, bucket-value listing, and PromQL evaluation.
use crate::api::dto::{
BucketValueDto, LogEntryDto, MetricDeclarationDto, ParamDeclarationDto,
StatsDto, TimelineBucketDto,
BucketValueDto, LogEntryDto, MetricDeclarationDto, ParamDeclarationDto, StatsDto,
TimelineBucketDto,
};
use crate::consts::{DEFAULT_TIMELINE_BUCKET_MS, MERGE_WINDOW_MS};
use crate::engine::data_value::DataValue;
@@ -12,17 +12,17 @@ use crate::engine::message::{EngineQuery, EngineResponse};
use crate::engine::metric_primitive::{MetricPrimitive, Number};
use crate::engine::promql::{LabelMatcher, PromqlQuery};
use crate::engine::rate;
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::snapshot::filter::CompiledFilter;
#[cfg(test)]
use crate::engine::snapshot::sort::SortHelperIndexEntry;
use crate::engine::snapshot::sort::{
log_entry_params, CompiledParamDeclaration, SortHelperAccess, SortHelperKey,
CompiledParamDeclaration, SortHelperAccess, SortHelperKey, log_entry_params,
};
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::timestamp::Timestamp;
use std::collections::BTreeMap;
#[cfg(test)]
use std::collections::HashMap;
use std::collections::BTreeMap;
#[cfg(test)]
pub(crate) fn handle_query(
@@ -30,7 +30,11 @@ pub(crate) fn handle_query(
query: EngineQuery,
sort_helper_indexes: &mut HashMap<String, SortHelperIndexEntry>,
) -> EngineResponse {
handle_query_inner(snapshot, query, SortHelperAccess::Mutable(sort_helper_indexes))
handle_query_inner(
snapshot,
query,
SortHelperAccess::Mutable(sort_helper_indexes),
)
}
/// Compiles `filter` against `snapshot`, or returns `Ok(None)` when there's no filter string
@@ -41,7 +45,9 @@ fn compile_optional_filter(
filter: Option<&str>,
now: Timestamp,
) -> Result<Option<CompiledFilter>, String> {
filter.map(|filter| snapshot.compile_filter_query(filter, now)).transpose()
filter
.map(|filter| snapshot.compile_filter_query(filter, now))
.transpose()
}
/// Walks the sort-helper index for `sort_by` — rather than `snapshot.logs_range()`, which is
@@ -67,7 +73,9 @@ fn logs_sorted_by_param(
let index = &snapshot.index;
static EMPTY_HELPER: std::sync::LazyLock<BTreeMap<SortHelperKey, Timestamp>> =
std::sync::LazyLock::new(BTreeMap::new);
let helper_index = sort_helper_access.resolve(index, sort_by).unwrap_or(&EMPTY_HELPER);
let helper_index = sort_helper_access
.resolve(index, sort_by)
.unwrap_or(&EMPTY_HELPER);
let mut entries = Vec::new();
for (helper_key, timestamp) in helper_index.iter() {
@@ -86,9 +94,10 @@ fn logs_sorted_by_param(
{
continue;
}
let Some(DataValue::LogLiteral { text }) =
index.get(&IndexKey::IngestedLog { sequence, timestamp: *timestamp })
else {
let Some(DataValue::LogLiteral { text }) = index.get(&IndexKey::IngestedLog {
sequence,
timestamp: *timestamp,
}) else {
continue;
};
if !compiled_filter.is_none_or(|filter| filter.eval(sequence, *timestamp, text)) {
@@ -117,14 +126,17 @@ pub(super) fn handle_query_inner(
match query {
EngineQuery::Logs(params) => {
let compiled_filter =
match compile_optional_filter(snapshot, params.filter.as_deref(), Timestamp::now()) {
match compile_optional_filter(snapshot, params.filter.as_deref(), Timestamp::now())
{
Ok(compiled) => compiled,
Err(err) => return EngineResponse::Error(err),
};
if let Some(sort_by) = params.sort_by.as_deref() {
if !snapshot.param_declarations_range().any(|(name, ..)| name == sort_by) {
return EngineResponse::Error(format!("unknown param '{sort_by}' for sort_by"));
}
if let Some(sort_by) = params.sort_by.as_deref()
&& !snapshot
.param_declarations_range()
.any(|(name, ..)| name == sort_by)
{
return EngineResponse::Error(format!("unknown param '{sort_by}' for sort_by"));
}
let limit = params.limit.unwrap_or(500).min(10_000);
let compiled_params = snapshot.compiled_param_declarations();
@@ -135,7 +147,9 @@ pub(super) fn handle_query_inner(
let filtered = snapshot
.logs_in_sequence_range((since_bound, std::ops::Bound::Unbounded))
.filter(|(_, timestamp, _)| {
params.start_ms.is_none_or(|start| timestamp.as_millis() >= start)
params
.start_ms
.is_none_or(|start| timestamp.as_millis() >= start)
&& params.end_ms.is_none_or(|end| timestamp.as_millis() <= end)
})
.filter(|(sequence, timestamp, text)| {
@@ -172,14 +186,20 @@ pub(super) fn handle_query_inner(
}
EngineQuery::Timeline(params) => {
let compiled_filter =
match compile_optional_filter(snapshot, params.filter.as_deref(), Timestamp::now()) {
match compile_optional_filter(snapshot, params.filter.as_deref(), Timestamp::now())
{
Ok(compiled) => compiled,
Err(err) => return EngineResponse::Error(err),
};
let bucket_ms = params.bucket_ms.unwrap_or(DEFAULT_TIMELINE_BUCKET_MS).max(1);
let bucket_ms = params
.bucket_ms
.unwrap_or(DEFAULT_TIMELINE_BUCKET_MS)
.max(1);
let mut buckets: BTreeMap<i64, u64> = BTreeMap::new();
for (sequence, timestamp, text) in snapshot.logs_range().filter(|(_, timestamp, _)| {
params.start_ms.is_none_or(|start| timestamp.as_millis() >= start)
params
.start_ms
.is_none_or(|start| timestamp.as_millis() >= start)
&& params.end_ms.is_none_or(|end| timestamp.as_millis() <= end)
}) {
if compiled_filter
@@ -221,34 +241,56 @@ pub(super) fn handle_query_inner(
})
}
EngineQuery::ParamDeclarations => {
let declarations = snapshot.param_declarations_range()
.map(|(name, message_match_regex, capture_group, json_field_path, kind, elapsed_since_last, auto_derived, _)| {
ParamDeclarationDto {
name: name.to_string(),
message_match_regex: message_match_regex.to_string(),
capture_group: capture_group.to_string(),
json_field_path: json_field_path.map(|path| path.to_string()),
let declarations = snapshot
.param_declarations_range()
.map(
|(
name,
message_match_regex,
capture_group,
json_field_path,
kind,
elapsed_since_last,
auto_derived,
reserved: false,
}
})
_,
)| {
ParamDeclarationDto {
name: name.to_string(),
message_match_regex: message_match_regex.to_string(),
capture_group: capture_group.to_string(),
json_field_path: json_field_path.map(|path| path.to_string()),
kind,
elapsed_since_last,
auto_derived,
reserved: false,
}
},
)
.collect();
EngineResponse::ParamDeclarations(declarations)
}
EngineQuery::MetricDeclarations => {
let declarations = snapshot.metric_declarations_range()
.map(|(name, message_match_regex, label_params, value_param, matched_labels, unmatched_labels)| {
MetricDeclarationDto {
name: name.to_string(),
message_match_regex: message_match_regex.to_string(),
label_params: label_params.to_vec(),
value_param: value_param.map(|s| s.to_string()),
matched_labels: matched_labels.cloned(),
unmatched_labels: unmatched_labels.cloned(),
}
})
let declarations = snapshot
.metric_declarations_range()
.map(
|(
name,
message_match_regex,
label_params,
value_param,
matched_labels,
unmatched_labels,
)| {
MetricDeclarationDto {
name: name.to_string(),
message_match_regex: message_match_regex.to_string(),
label_params: label_params.to_vec(),
value_param: value_param.map(|s| s.to_string()),
matched_labels: matched_labels.cloned(),
unmatched_labels: unmatched_labels.cloned(),
}
},
)
.collect();
EngineResponse::MetricDeclarations(declarations)
}
@@ -261,7 +303,8 @@ pub(super) fn handle_query_inner(
limit,
} => {
let limit = limit.unwrap_or(500).min(10_000);
let values: Vec<BucketValueDto> = snapshot.bucket_values(&metric_name, None)
let values: Vec<BucketValueDto> = snapshot
.bucket_values(&metric_name, None)
.filter(|(labels, _)| {
labels_filter.as_ref().is_none_or(|filter| {
filter.iter().all(|(name, expected)| {
@@ -286,7 +329,11 @@ pub(super) fn handle_query_inner(
.collect();
EngineResponse::BucketValues(values)
}
EngineQuery::Promql { query, start_ms, end_ms } => {
EngineQuery::Promql {
query,
start_ms,
end_ms,
} => {
let response = PromqlQuery::parse(&query)
.and_then(|parsed| snapshot.evaluate_promql(&parsed, start_ms, end_ms));
EngineResponse::Promql(response)
@@ -318,7 +365,9 @@ pub(crate) fn matches_label_matchers(
matchers: &[LabelMatcher],
) -> bool {
matchers.iter().all(|matcher| {
labels.get(&matcher.name).is_some_and(|value| value.to_string() == matcher.value)
labels
.get(&matcher.name)
.is_some_and(|value| value.to_string() == matcher.value)
})
}
@@ -338,29 +387,34 @@ pub(crate) fn windowed_rate_series(
start_ms: Option<i64>,
end_ms: Option<i64>,
) -> Vec<(i64, f64)> {
let ts_points: Vec<(Timestamp, f64)> =
points.iter().map(|(ms, value)| (Timestamp::from_millis(*ms), *value)).collect();
let ts_points: Vec<(Timestamp, f64)> = points
.iter()
.map(|(ms, value)| (Timestamp::from_millis(*ms), *value))
.collect();
let mut series: Vec<(i64, f64)> = rate::windowed_rate_series(&ts_points, range_ms)
.into_iter()
.map(|(ts, value)| (ts.as_millis(), value))
.collect();
if let Some(end) = end_ms {
if series.last().is_none_or(|(ts, _)| *ts < end) {
let value = rate::rate_at(&ts_points, Timestamp::from_millis(end), range_ms);
series.push((end, value));
}
if let Some(end) = end_ms
&& series.last().is_none_or(|(ts, _)| *ts < end)
{
let value = rate::rate_at(&ts_points, Timestamp::from_millis(end), range_ms);
series.push((end, value));
}
if let Some(start) = start_ms {
if series.first().is_none_or(|(ts, _)| *ts > start) {
let value = rate::rate_at(&ts_points, Timestamp::from_millis(start), range_ms);
series.insert(0, (start, value));
}
if let Some(start) = start_ms
&& series.first().is_none_or(|(ts, _)| *ts > start)
{
let value = rate::rate_at(&ts_points, Timestamp::from_millis(start), range_ms);
series.insert(0, (start, value));
}
series
}
pub(crate) fn merge_series_windowed(sub_series: &[Vec<(i64, f64)>], is_avg: bool) -> Vec<(i64, f64)> {
pub(crate) fn merge_series_windowed(
sub_series: &[Vec<(i64, f64)>],
is_avg: bool,
) -> Vec<(i64, f64)> {
if sub_series.len() <= 1 {
return sub_series.first().cloned().unwrap_or_default();
}
@@ -375,7 +429,11 @@ pub(crate) fn merge_series_windowed(sub_series: &[Vec<(i64, f64)>], is_avg: bool
.into_iter()
.map(|(timestamp_ms, values)| {
let sum: f64 = values.iter().sum();
let value = if is_avg { sum / values.len() as f64 } else { sum };
let value = if is_avg {
sum / values.len() as f64
} else {
sum
};
(timestamp_ms, value)
})
.collect()
@@ -398,7 +456,11 @@ pub(crate) fn merge_series_locf(sub_series: &[Vec<(i64, f64)>], is_avg: bool) ->
}
}
let sum: f64 = last_values.iter().sum();
let value = if is_avg { sum / last_values.len() as f64 } else { sum };
let value = if is_avg {
sum / last_values.len() as f64
} else {
sum
};
(ts, value)
})
.collect()
src/engine/snapshot/sort.rs
+37 -13
View File
@@ -4,12 +4,12 @@
//! kept incrementally up to date on ingest, rather than rebuilt per query.
use crate::engine::data_value::DataValue;
use crate::engine::index_key::IndexKey;
use crate::engine::data_value::MetricPrimitiveKind;
use crate::engine::index_key::IndexKey;
use crate::engine::metric_primitive::MetricPrimitive;
use crate::engine::snapshot::metrics::parse_metric_primitive;
#[cfg(test)]
use crate::engine::snapshot::EngineSnapshot;
use crate::engine::snapshot::metrics::parse_metric_primitive;
use crate::engine::timestamp::Timestamp;
use regex::Regex;
use std::collections::{BTreeMap, HashMap};
@@ -26,7 +26,14 @@ pub(crate) struct SortHelperIndexEntry {
pub(crate) last_accessed: std::time::Instant,
}
pub(super) type CompiledParamDeclaration = (String, Regex, String, Option<String>, MetricPrimitiveKind, bool);
pub(super) type CompiledParamDeclaration = (
String,
Regex,
String,
Option<String>,
MetricPrimitiveKind,
bool,
);
pub(crate) fn derive_single_param(
text: &str,
@@ -49,7 +56,10 @@ pub(crate) fn derive_single_param(
parse_metric_primitive(&raw, *kind)
}
pub(crate) fn derive_log_params(text: &str, declarations: &[CompiledParamDeclaration]) -> BTreeMap<String, MetricPrimitive> {
pub(crate) fn derive_log_params(
text: &str,
declarations: &[CompiledParamDeclaration],
) -> BTreeMap<String, MetricPrimitive> {
let mut params = BTreeMap::new();
for (name, regex, capture_group, json_field_path, kind, log_stream_gated) in declarations {
if *log_stream_gated && !crate::engine::logfmt::is_log_stream_line(text) {
@@ -63,10 +73,12 @@ pub(crate) fn derive_log_params(text: &str, declarations: &[CompiledParamDeclara
continue;
};
let raw: String = match json_field_path.as_deref().filter(|path| !path.is_empty()) {
Some(field_path) => match crate::engine::data_value::extract_json_field(text, field_path) {
Some(value) => value,
None => continue,
},
Some(field_path) => {
match crate::engine::data_value::extract_json_field(text, field_path) {
Some(value) => value,
None => continue,
}
}
None => match captures.name(capture_group) {
Some(m) => m.as_str().to_string(),
None => continue,
@@ -80,9 +92,15 @@ pub(crate) fn derive_log_params(text: &str, declarations: &[CompiledParamDeclara
params
}
pub(super) fn log_entry_params(text: &str, declarations: &[CompiledParamDeclaration]) -> BTreeMap<String, MetricPrimitive> {
pub(super) fn log_entry_params(
text: &str,
declarations: &[CompiledParamDeclaration],
) -> BTreeMap<String, MetricPrimitive> {
let mut params = derive_log_params(text, declarations);
params.insert("ottyl_fmt".to_string(), MetricPrimitive::String(crate::engine::logfmt::detect_format(text).to_string()));
params.insert(
"ottyl_fmt".to_string(),
MetricPrimitive::String(crate::engine::logfmt::detect_format(text).to_string()),
);
params
}
@@ -103,13 +121,19 @@ impl<'a> SortHelperAccess<'a> {
#[cfg(test)]
SortHelperAccess::Mutable(map) => {
if !map.contains_key(sort_by) {
let built = EngineSnapshot::from_index(index.clone()).build_sort_helper_index(sort_by);
let built =
EngineSnapshot::from_index(index.clone()).build_sort_helper_index(sort_by);
map.insert(
sort_by.to_string(),
SortHelperIndexEntry { index: built, last_accessed: std::time::Instant::now() },
SortHelperIndexEntry {
index: built,
last_accessed: std::time::Instant::now(),
},
);
}
let entry = map.get_mut(sort_by).expect("just inserted above if missing");
let entry = map
.get_mut(sort_by)
.expect("just inserted above if missing");
entry.last_accessed = std::time::Instant::now();
map.get(sort_by).map(|entry| &entry.index)
}
src/engine/tests.rs
+2790 -2091
View File
File diff suppressed because it is too large Load Diff
tickets/094-adopt-rustfmt.md
-34
View File
@@ -1,34 +0,0 @@
# 094 — Adopt rustfmt and format the codebase
## Summary
The Rust backend has no `rustfmt.toml` and isn't currently formatted
to rustfmt's defaults: `cargo fmt -- --check` exits non-zero, reporting
diffs across 24 files (352 diff hunks total).
## Where
- No `rustfmt.toml` / `.rustfmt.toml` exists at the repo root.
- `cargo fmt -- --check` reports diffs in `src/api/{dto,mcp_server,mod}.rs`,
`src/cli/{mod,sessions}.rs`, `src/config/{mod,profile}.rs`, and most
of `src/engine/**` (`alerts.rs`, `data_value.rs`, `filter_datetime.rs`,
`filter_parser.rs`, `ingest.rs`, `logfmt.rs`, `message.rs`, `mod.rs`,
`promql.rs`, `rate.rs`, `snapshot/{filter,metrics,mod,params,query,sort}.rs`,
`tests.rs`). Heaviest: `src/engine/snapshot/mod.rs` (~65 hunks),
`src/config/profile.rs` (~38), `src/api/mod.rs` (~35),
`src/engine/alerts.rs` (~28).
- `Makefile:26` (`lint-be`) currently only runs `cargo clippy
--all-targets` — no `cargo fmt --check` step.
## Scope
- Add a `rustfmt.toml` if any defaults should be overridden (otherwise
an empty/absent file is fine and rustfmt defaults apply — decide
whether to keep the file empty for documentation purposes or omit
it).
- Run `cargo fmt` across the whole `src/` tree as a single,
non-functional formatting commit (don't mix with logic changes, to
keep the diff reviewable and `git blame`-friendly).
- Add a `cargo fmt --check` step to `make lint` (or a new `make
fmt-check` wired into CI) so future changes can't drift out of
format again.
tickets/095-enforce-clippy-clean.md
-41
View File
@@ -1,41 +0,0 @@
# 095 — Resolve clippy warnings and make lint enforce a clean run
## Summary
`make lint`'s backend target runs `cargo clippy --all-targets` with no
`-D warnings` (or equivalent), so clippy warnings accumulate silently
instead of failing the lint step. There are currently 23 unique
warnings across 5 lint categories.
## Where
- `Makefile:26` (`lint-be:`) — `cargo clippy --all-targets`, no denial
flag.
- `cargo clippy --all-targets` currently reports, by lint:
- `clippy::cloned_ref_to_slice_refs` — 10 occurrences, mostly in
`src/engine/alerts.rs` test code (e.g. around lines 469, 472, 493,
534).
- `clippy::collapsible_if` — 8 occurrences, concentrated in
`src/engine/snapshot/mod.rs` (lines 608, 940, 961) and
`src/engine/alerts.rs` (around lines 510, 512).
- `clippy::type_complexity` — 3 occurrences.
- `clippy::single_match` — 1 occurrence.
- `clippy::unnecessary_get_then_check` — 1 occurrence,
`src/engine/tests.rs:2234`
(`entries[1].params.get("level").is_none()`).
## Scope
- Fix all 23 current warnings (mechanical for most: `from_ref` for the
slice-ref clone cases, collapsing nested `if`s, `is_none()` →
`!contains_key`/`get().is_none()` equivalent simplification per
clippy's suggestion, breaking up the complex types clippy flags or
adding a type alias, converting the single-arm `match` to `if let`).
- Once clean, change `Makefile:26` to deny warnings (e.g. `cargo
clippy --all-targets -- -D warnings`) so the target fails on any
future regression instead of just printing them.
- Don't add blanket `#[allow(...)]`s to silence these — fix the
underlying code per clippy's suggestion, matching the existing
`#[allow(dead_code)]` uses elsewhere in the codebase which are for
genuinely-unused-for-now items, not a precedent for suppressing
active lints.