65 lines
2.9 KiB
Markdown
65 lines
2.9 KiB
Markdown
# 073 — Track and surface ingestion-to-engine-processing lag (high watermark)
|
|
|
|
## Summary
|
|
|
|
Wishlist feedback, verbatim:
|
|
|
|
> There needs to be a ingstion to parsing latency on frontend.
|
|
> Basically when a log is read, it gets what would be the `ottyl_ts`
|
|
> value. However, the engine can lag on ingestion. That lag needs to
|
|
> be tracked as a high watermark. We can have a collection that stores
|
|
> say 5 highest values within the last 5 seconds. If a value in that
|
|
> collection is older than the 5s window, it gets replaced. If a value
|
|
> is higher than the minimum current, it gets replaced. The frontend
|
|
> will display the highest single value (humantime duration lag, or ms
|
|
> lag) alongside where it displays the logs and allocation size.
|
|
|
|
Confirmed the mechanism this is about: `ingest.rs` stamps
|
|
`timestamp: Timestamp::now()` at read time (this becomes `ottyl_ts`,
|
|
e.g. `ingest.rs:25`/`:107`) *before* sending `EngineMessage::Ingest`
|
|
over the mpsc channel to the engine actor task. The engine only
|
|
processes it whenever the actor's message loop gets scheduled to it —
|
|
under load (large backlog, slow query in flight, etc.) that gap can
|
|
grow. Today nothing measures `(engine processing time) - (ottyl_ts)`,
|
|
so there's no visibility into ingest backpressure.
|
|
|
|
The "allocation size" reference point is `approx_bytes_total`
|
|
(`src/dto.rs:80`, ticket 011), currently shown in
|
|
`webapp/src/components/NavMenu.tsx:104` in the stats popover — that's
|
|
the existing display location the lag figure should sit alongside.
|
|
|
|
## Scope
|
|
|
|
- In the engine actor's ingest handling (where `EngineMessage::Ingest`
|
|
is received, `engine.rs`), compute `lag = Timestamp::now() -
|
|
message.timestamp` at the moment of processing.
|
|
- Maintain a small rolling high-watermark collection per the spec
|
|
above: keep up to 5 entries, each valid only within the last 5
|
|
seconds; a new lag value replaces the current minimum-held entry if
|
|
it's larger, and any entry older than the 5s window is evicted
|
|
(replaced) regardless of magnitude. Expose the current max of that
|
|
collection.
|
|
- Add this figure to whatever stats DTO backs the popover
|
|
`approx_bytes_total` comes from, and to the corresponding stats
|
|
query/response path in `api.rs`.
|
|
- Display it in `NavMenu.tsx` next to `approx_bytes_total` (`:104`),
|
|
formatted as a humantime-style duration for larger lag and plain ms
|
|
for small values — match whatever precedent this codebase already
|
|
uses for duration formatting (check `webapp/src/lib` for an existing
|
|
humanize helper before writing a new one).
|
|
|
|
## Acceptance criteria
|
|
|
|
- Under normal (non-lagged) ingestion, the displayed value stays near
|
|
zero.
|
|
- Artificially delaying engine processing (e.g. a slow query
|
|
saturating the actor's message loop, or a synthetic test) causes the
|
|
displayed high-watermark to rise and correctly decays out of the
|
|
rolling 5s window afterward.
|
|
- The value is visible in the same UI location as
|
|
`approx_bytes_total`.
|
|
|
|
## Depends on
|
|
|
|
None.
|