61 lines
2.8 KiB
Markdown
61 lines
2.8 KiB
Markdown
# 075 — Investigate intermittent API unresponsiveness
|
|
|
|
## Summary
|
|
|
|
Wishlist feedback, verbatim: "The API is really unresponsive
|
|
sometimes. Create a ticket for me to investigate, not an agent.
|
|
Populate with some ideas."
|
|
|
|
This ticket is explicitly for Hugh to investigate by hand, not for an
|
|
agent to pick up and fix — it's a starting list of hypotheses, not a
|
|
scoped implementation task.
|
|
|
|
## Candidate causes to check
|
|
|
|
- **Single actor task, single mpsc channel.** All engine state lives
|
|
behind one tokio task (`EngineHandle`/`EngineMessage`,
|
|
`engine.rs`), and every HTTP request in `api.rs` becomes a message
|
|
send + oneshot await. A single slow query (e.g. a large unsorted
|
|
timeline materialize, or the pre-060 sort path if ticket 074 turns
|
|
out to still be hitting a full-scan fallback) blocks the entire actor
|
|
loop — every other in-flight request queues behind it, including
|
|
cheap ones like stats or param-list lookups. Worth instrumenting:
|
|
time each `EngineMessage` variant's handling and log/expose the
|
|
slowest ones.
|
|
- **Sort helper index builds are synchronous full scans.**
|
|
`build_sort_helper_index` (`engine.rs:1383`) does a one-time full
|
|
scan of every currently-ingested log the first time a param is
|
|
sorted by, inline in the actor's message handling — with a large log
|
|
count this could itself be the stall, and would look exactly like
|
|
"API goes unresponsive briefly" right after picking a new sort
|
|
column. Related to ticket 074.
|
|
- **Prometheus/OpenMetrics polling (`ingest.rs::run_metrics_poll`)**
|
|
runs concurrently with log ingestion feeding the same channel — check
|
|
whether a slow or hanging `--metrics-url` endpoint can back up the
|
|
shared `EngineMessage` channel and starve log-ingestion/API-serving
|
|
throughput.
|
|
- **Unbounded per-request work scaling with log volume.** Any handler
|
|
that iterates the full retained log index (dashboards/PromQL
|
|
evaluation, `promql.rs`; alert rule evaluation, `alerts.rs`) rather
|
|
than an indexed range could scale badly as `max logs in memory`
|
|
grows, especially with multiple dashboard panels polling
|
|
concurrently.
|
|
- **Frontend polling cadence.** `webapp/src/lib/polling.ts` — check
|
|
whether multiple open tabs/panels each poll independently at a short
|
|
interval, potentially self-inflicting load rather than the backend
|
|
being intrinsically slow.
|
|
- Worth capturing wall-clock timing (both server-side, and via browser
|
|
devtools network timing) the next time it's reproduced, to know
|
|
whether the stall is queueing (actor busy), compute (one query is
|
|
slow), or genuinely I/O (waiting on `--metrics-url` or file tailing).
|
|
|
|
## Scope
|
|
|
|
Investigation only — no code changes implied by this ticket. Once a
|
|
concrete cause is found, split into a targeted fix ticket.
|
|
|
|
## Depends on
|
|
|
|
None, but related to ticket 074 (sort helper index scans) and touches
|
|
the same actor architecture ticket 060/061 modified.
|