# 075 — Investigate intermittent API unresponsiveness

## Summary

Wishlist feedback, verbatim: "The API is really unresponsive
sometimes. Create a ticket for me to investigate, not an agent.
Populate with some ideas."

This ticket is explicitly for Hugh to investigate by hand, not for an
agent to pick up and fix — it's a starting list of hypotheses, not a
scoped implementation task.

## Candidate causes to check

- **Single actor task, single mpsc channel.** All engine state lives
  behind one tokio task (`EngineHandle`/`EngineMessage`,
  `engine.rs`), and every HTTP request in `api.rs` becomes a message
  send + oneshot await. A single slow query (e.g. a large unsorted
  timeline materialize, or the pre-060 sort path if ticket 074 turns
  out to still be hitting a full-scan fallback) blocks the entire actor
  loop — every other in-flight request queues behind it, including
  cheap ones like stats or param-list lookups. Worth instrumenting:
  time each `EngineMessage` variant's handling and log/expose the
  slowest ones.
- **Sort helper index builds are synchronous full scans.**
  `build_sort_helper_index` (`engine.rs:1383`) does a one-time full
  scan of every currently-ingested log the first time a param is
  sorted by, inline in the actor's message handling — with a large log
  count this could itself be the stall, and would look exactly like
  "API goes unresponsive briefly" right after picking a new sort
  column. Related to ticket 074.
- **Prometheus/OpenMetrics polling (`ingest.rs::run_metrics_poll`)**
  runs concurrently with log ingestion feeding the same channel — check
  whether a slow or hanging `--metrics-url` endpoint can back up the
  shared `EngineMessage` channel and starve log-ingestion/API-serving
  throughput.
- **Unbounded per-request work scaling with log volume.** Any handler
  that iterates the full retained log index (dashboards/PromQL
  evaluation, `promql.rs`; alert rule evaluation, `alerts.rs`) rather
  than an indexed range could scale badly as `max logs in memory`
  grows, especially with multiple dashboard panels polling
  concurrently.
- **Frontend polling cadence.** `webapp/src/lib/polling.ts` — check
  whether multiple open tabs/panels each poll independently at a short
  interval, potentially self-inflicting load rather than the backend
  being intrinsically slow.
- Worth capturing wall-clock timing (both server-side, and via browser
  devtools network timing) the next time it's reproduced, to know
  whether the stall is queueing (actor busy), compute (one query is
  slow), or genuinely I/O (waiting on `--metrics-url` or file tailing).

## Scope

Investigation only — no code changes implied by this ticket. Once a
concrete cause is found, split into a targeted fix ticket.

## Depends on

None, but related to ticket 074 (sort helper index scans) and touches
the same actor architecture ticket 060/061 modified.