@@ -1,3 +1,177 @@
# ottyl
Observability dashboard from terminal output
# Usage
At a high level, the idea is that a program will output logs, which can be captured and displayed on a timeline and dashboard similar to grafana.
``` bash
cli_command --verbose | ottyl
```
# Architecture
ottyl reads the logs via stdin and indexes it in-memory in a BTreeMap for property search.
#
This is the main program, hosting the webapp over axum.
The engine reads stdin and sends line-split messages to a log ingester.
The log ingester holds a mutex over a BTreeMap<IndexKey, DataValue>.
The BTreeMap may be a custom variation of the std library one, because we need to be able to scan specific ranges and jump to keys.
```
/// Get the given key, or the next key in the btree
fn get_or_next(&self, key: &K) -> Option<Entry<K, V>>{...}
/// Get the next key in the btree
fn get_or_next(&self, key: &K) -> Option<Entry<K, V>>{...}
/// Get the previous key in the btree
fn previous(&self, key: &K) -> Option<Entry<K, V>> {...}
```
IndexKey is how we will index the logs and metrics required for search and dashboards.
```
enum IndexKey {
// Convenient way to find the start of the range
// It should never be inserted into the btreemap, but it can be used for navigating it
// This is true for all position-marker type keys
Start,
// A key for a literal raw log, where sequence is effectively the line number
// and timestamp is the time at which it was ingested into ottyl
IngestedLog{sequence: usize, timestamp: Instant},
// Convenience to get the end of the ingested log range and start of param range
LogParamSplit,
// Param declarations allow the user to specific how a parameter for a message can be derived
// This is useful for deriving timestamps from logs, that are different from the ingested timestamp (ex cat log_dump.txt)
// Other metrics may also be derived, like log level etc
ParamDeclaration{name: String},
// End of param range, start of metric declaration range
ParamMetricSplit,
// Used for storing the exact metric definition, so buckets and parameters can be derived
MetricDeclaration{name: String},
// Used to split metric declarations from their buckets
MetricBucketSplit,
//
Bucket{metric_name: String, labels: BTreeMap<String, MetricPrimitive>}
// End of key range, nothing should ever come after this
End,
}
```
DataValue is how we have a convenient pairing for a single BTreeMap KV mapping.
It covers all the result types for the primary BTreeMap.
```
enum DataValue {
// Stores the raw ingested log
// Used in IngestedLog keys
LogLiteral{text: String},
// Stores a metric declaration
// Used in MetricDeclaration keys
MetricDeclaration{message_match_regex: String, params: Vec<RegexToMetricPrimitiveMatcher>
// Stores the values of a bucket
// Used in Bucket keys
BucketValue{values: Vec<(Sequence, Timestamp, MetricPrimitive)>}
}
```
MetricPrimitive is literally just a simple way to track values without using json.
```
enum MetricPrimitive {
String(String),
Number(Number),
Timestamp(Instant),
}
```
# WebApp
ottyl provides a spa webapp dashboard in typescript react and tailwind.
It is served from an in-process axum binding.
``` mermaid
block
columns 2
NavMenu
block
columns 1
FilterBar
LogTimeline
LogPreview
Metrics
Dashboards
end
```
## NavMenu
The nav menu is largely for managing settings and presets.
Presets track how to parse and index logs, and stored dashboards.
That means that restarting the application for an CLI app does not lose dashboards and parsing rules.
Settings involve colour schemes, or declaring the path for the css.
It should be possible to import a css, so it gets stored in the same location that the profiles and settings are stored.
It should be possible to delete imported css.
Not all css needs to be imported, linking to some path or url should be viable also - those are not deletable.
Settings "owns" the profiles, so changing a settings file changes the used profiles.
At runtime, there is only 1 settings.
It can be set using clap cli args.
The settings file by default is stored in a sensible user location for cli settings.
## FilterBar
Syntax aware text search.
A parser can be written in pest.
For now, the syntax should support regex searching and metrics searching, such as `metric_name[param]<5` .
LogSequence ('sequence') and Timestamp ('timestamp') are reserved metrics.
```
regex("error") and sequence<5 or timestamp>12:00
```
DateTime parsing should support several formats
- relative humantime (`5h` , `-3m` etc)
- absolute
- fulldate
## LogTimeline
This should work similar to how elasticsearch has a log timeline.
Its a timeline of a barchart, where each bar is effectively some bucket of messages.
It reflects the selected time range.
Dragging over the log timeline displays a rectangle as it is happening and also the time range.
Once the drag stops, the selected range becomes the new filter time range.
## LogPreview
Log preview is simply a scrollable table of the log messages, like in kibana.
The sequence, and timestamp are visible.
It should be possible to pick which parameters are visible.
## Metrics
Metrics can be declared from logs.
There should be a language syntax for declaring metircs.
Once declared, already-ingested logs get indexed on the metric, and new metrics get ingested as well.
## Dashboards
Dashboards work the same way as in grafana.
There should be support for promql queries to derive the visualisation.
Supported charts should be bar chart, line chart, and pie chart.
Hugh Kaznowski