ottyl/tickets/075-investigate-api-unresponsiveness.md

075 — Investigate intermittent API unresponsiveness

Summary

Wishlist feedback, verbatim: "The API is really unresponsive sometimes. Create a ticket for me to investigate, not an agent. Populate with some ideas."

This ticket is explicitly for Hugh to investigate by hand, not for an agent to pick up and fix — it's a starting list of hypotheses, not a scoped implementation task.

Candidate causes to check

• Single actor task, single mpsc channel. All engine state lives behind one tokio task (EngineHandle/EngineMessage, engine.rs), and every HTTP request in api.rs becomes a message send + oneshot await. A single slow query (e.g. a large unsorted timeline materialize, or the pre-060 sort path if ticket 074 turns out to still be hitting a full-scan fallback) blocks the entire actor loop — every other in-flight request queues behind it, including cheap ones like stats or param-list lookups. Worth instrumenting: time each EngineMessage variant's handling and log/expose the slowest ones.
• Sort helper index builds are synchronous full scans. build_sort_helper_index (engine.rs:1383) does a one-time full scan of every currently-ingested log the first time a param is sorted by, inline in the actor's message handling — with a large log count this could itself be the stall, and would look exactly like "API goes unresponsive briefly" right after picking a new sort column. Related to ticket 074.
• Prometheus/OpenMetrics polling (ingest.rs::run_metrics_poll) runs concurrently with log ingestion feeding the same channel — check whether a slow or hanging --metrics-url endpoint can back up the shared EngineMessage channel and starve log-ingestion/API-serving throughput.
• Unbounded per-request work scaling with log volume. Any handler that iterates the full retained log index (dashboards/PromQL evaluation, promql.rs; alert rule evaluation, alerts.rs) rather than an indexed range could scale badly as max logs in memory grows, especially with multiple dashboard panels polling concurrently.
• Frontend polling cadence. webapp/src/lib/polling.ts — check whether multiple open tabs/panels each poll independently at a short interval, potentially self-inflicting load rather than the backend being intrinsically slow.
• Worth capturing wall-clock timing (both server-side, and via browser devtools network timing) the next time it's reproduced, to know whether the stall is queueing (actor busy), compute (one query is slow), or genuinely I/O (waiting on --metrics-url or file tailing).

Scope

Investigation only — no code changes implied by this ticket. Once a concrete cause is found, split into a targeted fix ticket.

Depends on

None, but related to ticket 074 (sort helper index scans) and touches the same actor architecture ticket 060/061 modified.