Left-right single-writer, multi-reader index

2026-09-06 16:05:04 +01:00
parent 9fc7540a5b
commit 1c635bc530
9 changed files with 684 additions and 203 deletions
@@ -1,60 +0,0 @@
-# 075 — Investigate intermittent API unresponsiveness
-
-## Summary
-
-Wishlist feedback, verbatim: "The API is really unresponsive
-sometimes. Create a ticket for me to investigate, not an agent.
-Populate with some ideas."
-
-This ticket is explicitly for Hugh to investigate by hand, not for an
-agent to pick up and fix — it's a starting list of hypotheses, not a
-scoped implementation task.
-
-## Candidate causes to check
-
- **Single actor task, single mpsc channel.** All engine state lives
-  behind one tokio task (`EngineHandle`/`EngineMessage`,
-  `engine.rs`), and every HTTP request in `api.rs` becomes a message
-  send + oneshot await. A single slow query (e.g. a large unsorted
-  timeline materialize, or the pre-060 sort path if ticket 074 turns
-  out to still be hitting a full-scan fallback) blocks the entire actor
-  loop — every other in-flight request queues behind it, including
-  cheap ones like stats or param-list lookups. Worth instrumenting:
-  time each `EngineMessage` variant's handling and log/expose the
-  slowest ones.
- **Sort helper index builds are synchronous full scans.**
-  `build_sort_helper_index` (`engine.rs:1383`) does a one-time full
-  scan of every currently-ingested log the first time a param is
-  sorted by, inline in the actor's message handling — with a large log
-  count this could itself be the stall, and would look exactly like
-  "API goes unresponsive briefly" right after picking a new sort
-  column. Related to ticket 074.
- **Prometheus/OpenMetrics polling (`ingest.rs::run_metrics_poll`)**
-  runs concurrently with log ingestion feeding the same channel — check
-  whether a slow or hanging `--metrics-url` endpoint can back up the
-  shared `EngineMessage` channel and starve log-ingestion/API-serving
-  throughput.
- **Unbounded per-request work scaling with log volume.** Any handler
-  that iterates the full retained log index (dashboards/PromQL
-  evaluation, `promql.rs`; alert rule evaluation, `alerts.rs`) rather
-  than an indexed range could scale badly as `max logs in memory`
-  grows, especially with multiple dashboard panels polling
-  concurrently.
- **Frontend polling cadence.** `webapp/src/lib/polling.ts` — check
-  whether multiple open tabs/panels each poll independently at a short
-  interval, potentially self-inflicting load rather than the backend
-  being intrinsically slow.
- Worth capturing wall-clock timing (both server-side, and via browser
-  devtools network timing) the next time it's reproduced, to know
-  whether the stall is queueing (actor busy), compute (one query is
-  slow), or genuinely I/O (waiting on `--metrics-url` or file tailing).
-
-## Scope
-
-Investigation only — no code changes implied by this ticket. Once a
-concrete cause is found, split into a targeted fix ticket.
-
-## Depends on
-
-None, but related to ticket 074 (sort helper index scans) and touches
-the same actor architecture ticket 060/061 modified.