ottyl/README.md

ottyl

Observability dashboard from terminal output

Usage

At a high level, the idea is that a program will output logs, which can be captured and displayed on a timeline and dashboard similar to grafana.

cli_command --verbose | ottyl

Architecture

ottyl reads the logs via stdin and indexes it in-memory in a BTreeMap for property search.

# Engine

This is the main program, hosting the webapp over axum.

The engine reads stdin and sends line-split messages to a log ingester. The log ingester holds a BTreeMap<IndexKey, DataValue>.

IndexKey is how we will index the logs and metrics required for search and dashboards.

enum IndexKey {
  // Convenient way to find the start of the range
  // It should never be inserted into the btreemap, but it can be used for navigating it
  // This is true for all position-marker type keys
  Start,
  // A key for a literal raw log, where sequence is effectively the line number
  // and timestamp is the time at which it was ingested into ottyl
  IngestedLog{sequence: usize, timestamp: Instant},
  // Convenience to get the end of the ingested log range and start of param range
  LogParamSplit,
  // Param declarations allow the user to specific how a parameter for a message can be derived
  // This is useful for deriving timestamps from logs, that are different from the ingested timestamp (ex cat log_dump.txt)
  // Other metrics may also be derived, like log level etc
  ParamDeclaration{name: String},
  // End of param range, start of metric declaration range
  ParamMetricSplit,
  // Used for storing the exact metric definition, so buckets and parameters can be derived
  MetricDeclaration{name: String},
  // Used to split metric declarations from their buckets
  MetricBucketSplit,
  // 
  Bucket{metric_name: String, labels: BTreeMap<String, MetricPrimitive>}
  // End of key range, nothing should ever come after this
  End,
}

DataValue is how we have a convenient pairing for a single BTreeMap KV mapping. It covers all the result types for the primary BTreeMap.

enum DataValue {
  // Stores the raw ingested log
  // Used in IngestedLog keys
  LogLiteral{text: String},
  // Stores a metric declaration
  // Used in MetricDeclaration keys
  MetricDeclaration{message_match_regex: String, params: Vec<RegexToMetricPrimitiveMatcher>
  // Stores the values of a bucket
  // Used in Bucket keys
  BucketValue{values: Vec<(Sequence, Timestamp, MetricPrimitive)>}
}

MetricPrimitive is literally just a simple way to track values without using json.

enum MetricPrimitive {
  String(String),
  Number(Number),
  Timestamp(Instant),
}

Tasks

The ingestion tasks reads stdin, sequences the logs, timestamps them and sends over a channel to the engine task.

The engine task processes incoming logs and incoming requests. It should be lock-free and the sole-owner of the btreemap.

Axum is not a specific task, but exists and serves the API and frontend. API requests get validated (like, correct syntax etc etc) and then requests are sent over a channel to the engine. The request includes a byte buffer, for response serialisation to go into it. The engine does the serialisation, because it is the sole owner of the btreemap and should not be copying memory simply to serialise it later. A callback is given, so the API knows the request is processed and response is ready to send back from the byte buffer. Rkyv can be used if that helps, otherwise sending raw bytes is fine as well.

WebApp

ottyl provides a spa webapp dashboard in typescript react and tailwind. It is served from an in-process axum binding.

block
columns 2
  NavMenu
  block
  columns 1
    FilterBar
    LogTimeline
    LogPreview
    Metrics
    Dashboards
  end

NavMenu

The nav menu is largely for managing settings and presets.

Presets track how to parse and index logs, and stored dashboards. That means that restarting the application for an CLI app does not lose dashboards and parsing rules.

Settings involve colour schemes, or declaring the path for the css. It should be possible to import a css, so it gets stored in the same location that the profiles and settings are stored. It should be possible to delete imported css. Not all css needs to be imported, linking to some path or url should be viable also - those are not deletable.

Settings "owns" the profiles, so changing a settings file changes the used profiles. At runtime, there is only 1 settings. It can be set using clap cli args.

The settings file by default is stored in a sensible user location for cli settings.

FilterBar

Syntax aware text search. A parser can be written in pest. For now, the syntax should support regex searching and metrics searching, such as metric_name[param]<5. LogSequence ('sequence') and Timestamp ('timestamp') are reserved metrics.

regex("error") and sequence<5 or timestamp>12:00

DateTime parsing should support several formats

• relative humantime (5h, -3m etc)
• absolute
• fulldate

LogTimeline

This should work similar to how elasticsearch has a log timeline. Its a timeline of a barchart, where each bar is effectively some bucket of messages. It reflects the selected time range. Dragging over the log timeline displays a rectangle as it is happening and also the time range. Once the drag stops, the selected range becomes the new filter time range.

LogPreview

Log preview is simply a scrollable table of the log messages, like in kibana. The sequence, and timestamp are visible. It should be possible to pick which parameters are visible.

Metrics

Metrics can be declared from logs. There should be a language syntax for declaring metircs.

Once declared, already-ingested logs get indexed on the metric, and new metrics get ingested as well.

Dashboards

Dashboards work the same way as in grafana. There should be support for promql queries to derive the visualisation. Supported charts should be bar chart, line chart, and pie chart.

Ideas

This part describes ideas that can be implemented in the future, but not now.

Ntfy.sh alerts

Have matchers you are interested in. Configure the ntfy.sh channel. Send to the channel when an event triggers.

Sequence diagram of events

It would be awesome if it were possible to declare participants of a sequence diagram using parameters or matchers. Example node_id for a cluster test or something. Then it may be possible to capture specific events of interest. There wouldn't be a relation between nodes, only the sequence of events determines relations in the sequence diagram.