ottyl/tickets/073-ingestion-to-engine-processing-lag-tracking.md

073 — Track and surface ingestion-to-engine-processing lag (high watermark)

Summary

Wishlist feedback, verbatim:

There needs to be a ingstion to parsing latency on frontend. Basically when a log is read, it gets what would be the ottyl_ts value. However, the engine can lag on ingestion. That lag needs to be tracked as a high watermark. We can have a collection that stores say 5 highest values within the last 5 seconds. If a value in that collection is older than the 5s window, it gets replaced. If a value is higher than the minimum current, it gets replaced. The frontend will display the highest single value (humantime duration lag, or ms lag) alongside where it displays the logs and allocation size.

Confirmed the mechanism this is about: ingest.rs stamps timestamp: Timestamp::now() at read time (this becomes ottyl_ts, e.g. ingest.rs:25/:107) before sending EngineMessage::Ingest over the mpsc channel to the engine actor task. The engine only processes it whenever the actor's message loop gets scheduled to it — under load (large backlog, slow query in flight, etc.) that gap can grow. Today nothing measures (engine processing time) - (ottyl_ts), so there's no visibility into ingest backpressure.

The "allocation size" reference point is approx_bytes_total (src/dto.rs:80, ticket 011), currently shown in webapp/src/components/NavMenu.tsx:104 in the stats popover — that's the existing display location the lag figure should sit alongside.

Scope

• In the engine actor's ingest handling (where EngineMessage::Ingest is received, engine.rs), compute lag = Timestamp::now() - message.timestamp at the moment of processing.
• Maintain a small rolling high-watermark collection per the spec above: keep up to 5 entries, each valid only within the last 5 seconds; a new lag value replaces the current minimum-held entry if it's larger, and any entry older than the 5s window is evicted (replaced) regardless of magnitude. Expose the current max of that collection.
• Add this figure to whatever stats DTO backs the popover approx_bytes_total comes from, and to the corresponding stats query/response path in api.rs.
• Display it in NavMenu.tsx next to approx_bytes_total (:104), formatted as a humantime-style duration for larger lag and plain ms for small values — match whatever precedent this codebase already uses for duration formatting (check webapp/src/lib for an existing humanize helper before writing a new one).

Acceptance criteria

• Under normal (non-lagged) ingestion, the displayed value stays near zero.
• Artificially delaying engine processing (e.g. a slow query saturating the actor's message loop, or a synthetic test) causes the displayed high-watermark to rise and correctly decays out of the rolling 5s window afterward.
• The value is visible in the same UI location as approx_bytes_total.

Depends on

None.