Update skill and tickets

tickets/073-ingestion-to-engine-processing-lag-tracking.md

@@ -0,0 +1,64 @@
+# 073 — Track and surface ingestion-to-engine-processing lag (high watermark)
+
+## Summary
+
+Wishlist feedback, verbatim:
+
+> There needs to be a ingstion to parsing latency on frontend.
+> Basically when a log is read, it gets what would be the `ottyl_ts`
+> value. However, the engine can lag on ingestion. That lag needs to
+> be tracked as a high watermark. We can have a collection that stores
+> say 5 highest values within the last 5 seconds. If a value in that
+> collection is older than the 5s window, it gets replaced. If a value
+> is higher than the minimum current, it gets replaced. The frontend
+> will display the highest single value (humantime duration lag, or ms
+> lag) alongside where it displays the logs and allocation size.
+
+Confirmed the mechanism this is about: `ingest.rs` stamps
+`timestamp: Timestamp::now()` at read time (this becomes `ottyl_ts`,
+e.g. `ingest.rs:25`/`:107`) *before* sending `EngineMessage::Ingest`
+over the mpsc channel to the engine actor task. The engine only
+processes it whenever the actor's message loop gets scheduled to it —
+under load (large backlog, slow query in flight, etc.) that gap can
+grow. Today nothing measures `(engine processing time) - (ottyl_ts)`,
+so there's no visibility into ingest backpressure.
+
+The "allocation size" reference point is `approx_bytes_total`
+(`src/dto.rs:80`, ticket 011), currently shown in
+`webapp/src/components/NavMenu.tsx:104` in the stats popover — that's
+the existing display location the lag figure should sit alongside.
+
+## Scope
+
+- In the engine actor's ingest handling (where `EngineMessage::Ingest`
+  is received, `engine.rs`), compute `lag = Timestamp::now() -
+  message.timestamp` at the moment of processing.
+- Maintain a small rolling high-watermark collection per the spec
+  above: keep up to 5 entries, each valid only within the last 5
+  seconds; a new lag value replaces the current minimum-held entry if
+  it's larger, and any entry older than the 5s window is evicted
+  (replaced) regardless of magnitude. Expose the current max of that
+  collection.
+- Add this figure to whatever stats DTO backs the popover
+  `approx_bytes_total` comes from, and to the corresponding stats
+  query/response path in `api.rs`.
+- Display it in `NavMenu.tsx` next to `approx_bytes_total` (`:104`),
+  formatted as a humantime-style duration for larger lag and plain ms
+  for small values — match whatever precedent this codebase already
+  uses for duration formatting (check `webapp/src/lib` for an existing
+  humanize helper before writing a new one).
+
+## Acceptance criteria
+
+- Under normal (non-lagged) ingestion, the displayed value stays near
+  zero.
+- Artificially delaying engine processing (e.g. a slow query
+  saturating the actor's message loop, or a synthetic test) causes the
+  displayed high-watermark to rise and correctly decays out of the
+  rolling 5s window afterward.
+- The value is visible in the same UI location as
+  `approx_bytes_total`.
+
+## Depends on
+
+None.