Update README
This commit is contained in:
Hugh Kaznowski
@@ -1,226 +1,52 @@
|
||||
# ottyl
|
||||
|
||||
Observability dashboard from terminal output
|
||||
|
||||
# Usage
|
||||
|
||||
At a high level, the idea is that a program will output logs, which can be captured and displayed on a timeline and dashboard similar to grafana.
|
||||
Observability for any command's terminal output — no logging backend, no agents, no config to ship.
|
||||
|
||||
```bash
|
||||
cli_command --verbose | ottyl
|
||||
|
||||
your_app --verbose | ottyl
|
||||
```
|
||||
|
||||
# Building
|
||||
Pipe a program's stdout into `ottyl` and turn it into a searchable log timeline, ad-hoc metrics extracted from the log lines themselves, and Grafana-style dashboards — served from a single local binary, for the lifetime of that process:
|
||||
|
||||
The webapp is embedded into the `ottyl` binary at compile time, so `webapp/dist` must exist *before* running `cargo build`:
|
||||
- **Search:** A syntax-aware filter bar over your logs — regex matches and metric comparisons like `latency_ms[value]<200`, with relative (`5h`, `-3m`) and absolute datetimes.
|
||||
- **Log timeline:** A bucketed bar chart of message volume over time, Kibana-style. Drag across it to set the active time range.
|
||||
- **Metrics from logs:** Declare a metric by matching a regex against your log lines and extracting a value. It applies retroactively to everything already ingested, and to everything that follows.
|
||||
- **Dashboards:** Bar, line, and pie chart panels driven by PromQL-style queries over your declared metrics.
|
||||
- **Mixed sources:** Poll an existing Prometheus/OpenMetrics endpoint alongside log ingestion, in the same dashboard.
|
||||
- **Profiles:** Parsing rules and dashboards are saved per app, so they're there again next time that app runs through ottyl.
|
||||
|
||||
## Use cases
|
||||
|
||||
- Debugging a noisy CLI or long-running script, instead of scrolling a terminal buffer.
|
||||
- Getting metrics and charts out of a program that only ever prints text.
|
||||
- Local dashboards without standing up a metrics stack.
|
||||
- Combining scraped Prometheus metrics with log-derived ones during a debugging session.
|
||||
|
||||
## Get started
|
||||
|
||||
Requires Rust and Node.js. The webapp is built and embedded into the binary, so the result is a single self-contained executable.
|
||||
|
||||
```bash
|
||||
cd webapp && npm install && npm run build && cd ..
|
||||
cargo build --release
|
||||
git clone <this-repo>
|
||||
cd ottyl
|
||||
make build
|
||||
your_app --verbose | ottyl
|
||||
```
|
||||
|
||||
The resulting binary is self-contained — no `webapp/dist` directory or `--static-dir` flag is needed at runtime.
|
||||
|
||||
# Architecture
|
||||
|
||||
ottyl reads the logs via stdin and indexes it in-memory in a BTreeMap for property search.
|
||||
|
||||
# Engine
|
||||
|
||||
This is the main program, hosting the webapp over axum.
|
||||
|
||||
The engine reads stdin and sends line-split messages to a log ingester.
|
||||
The log ingester holds a BTreeMap<IndexKey, DataValue>.
|
||||
|
||||
IndexKey is how we will index the logs and metrics required for search and dashboards.
|
||||
```
|
||||
enum IndexKey {
|
||||
// Convenient way to find the start of the range
|
||||
// It should never be inserted into the btreemap, but it can be used for navigating it
|
||||
// This is true for all position-marker type keys
|
||||
Start,
|
||||
// A key for a literal raw log, where sequence is effectively the line number
|
||||
// and timestamp is the time at which it was ingested into ottyl
|
||||
IngestedLog{sequence: usize, timestamp: Instant},
|
||||
// Convenience to get the end of the ingested log range and start of param range
|
||||
LogParamSplit,
|
||||
// Param declarations allow the user to specific how a parameter for a message can be derived
|
||||
// This is useful for deriving timestamps from logs, that are different from the ingested timestamp (ex cat log_dump.txt)
|
||||
// Other metrics may also be derived, like log level etc
|
||||
ParamDeclaration{name: String},
|
||||
// End of param range, start of metric declaration range
|
||||
ParamMetricSplit,
|
||||
// Used for storing the exact metric definition, so buckets and parameters can be derived
|
||||
MetricDeclaration{name: String},
|
||||
// Used to split metric declarations from their buckets
|
||||
MetricBucketSplit,
|
||||
//
|
||||
Bucket{metric_name: String, labels: BTreeMap<String, MetricPrimitive>}
|
||||
// End of key range, nothing should ever come after this
|
||||
End,
|
||||
}
|
||||
Or point it at a log file instead of stdin, and mix in a Prometheus/OpenMetrics endpoint:
|
||||
|
||||
```bash
|
||||
ottyl --file /var/log/your_app.log --metrics-url http://localhost:9090/metrics
|
||||
```
|
||||
|
||||
DataValue is how we have a convenient pairing for a single BTreeMap KV mapping.
|
||||
It covers all the result types for the primary BTreeMap.
|
||||
ottyl prints the URL of the local web server it starts (an OS-assigned free port by default; pin one with `--port`). See `ottyl --help` for the rest, including where settings and profiles live (`--settings-path`).
|
||||
|
||||
Full documentation — filtering syntax, parameters, metrics, dashboards — is built into the app itself, under the nav menu's docs section.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
make dev # frontend dev server with hot reload, proxying to a separately running `cargo run` backend
|
||||
make test # backend + frontend tests
|
||||
make lint # clippy + oxlint
|
||||
```
|
||||
enum DataValue {
|
||||
// Stores the raw ingested log
|
||||
// Used in IngestedLog keys
|
||||
LogLiteral{text: String},
|
||||
// Stores a metric declaration
|
||||
// Used in MetricDeclaration keys
|
||||
MetricDeclaration{message_match_regex: String, params: Vec<RegexToMetricPrimitiveMatcher>
|
||||
// Stores the values of a bucket
|
||||
// Used in Bucket keys
|
||||
BucketValue{values: Vec<(Sequence, Timestamp, MetricPrimitive)>}
|
||||
}
|
||||
```
|
||||
|
||||
MetricPrimitive is literally just a simple way to track values without using json.
|
||||
```
|
||||
enum MetricPrimitive {
|
||||
String(String),
|
||||
Number(Number),
|
||||
Timestamp(Instant),
|
||||
}
|
||||
```
|
||||
|
||||
## Tasks
|
||||
|
||||
The ingestion tasks reads stdin, sequences the logs, timestamps them and sends over a channel to the engine task.
|
||||
|
||||
The engine task processes incoming logs and incoming requests.
|
||||
It should be lock-free and the sole-owner of the btreemap.
|
||||
|
||||
Axum is not a specific task, but exists and serves the API and frontend.
|
||||
API requests get validated (like, correct syntax etc etc) and then requests are sent over a channel to the engine.
|
||||
The request includes a byte buffer, for response serialisation to go into it.
|
||||
The engine does the serialisation, because it is the sole owner of the btreemap and should not be copying memory simply to serialise it later.
|
||||
A callback is given, so the API knows the request is processed and response is ready to send back from the byte buffer.
|
||||
Rkyv can be used if that helps, otherwise sending raw bytes is fine as well.
|
||||
|
||||
# WebApp
|
||||
|
||||
ottyl provides a spa webapp dashboard in typescript react and tailwind.
|
||||
It is served from an in-process axum binding.
|
||||
|
||||
```mermaid
|
||||
block
|
||||
columns 2
|
||||
NavMenu
|
||||
block
|
||||
columns 1
|
||||
FilterBar
|
||||
LogTimeline
|
||||
LogPreview
|
||||
Metrics
|
||||
Dashboards
|
||||
end
|
||||
```
|
||||
|
||||
## NavMenu
|
||||
|
||||
The nav menu is largely for managing settings and presets.
|
||||
|
||||
Presets track how to parse and index logs, and stored dashboards.
|
||||
That means that restarting the application for an CLI app does not lose dashboards and parsing rules.
|
||||
|
||||
Settings involve colour schemes, or declaring the path for the css.
|
||||
It should be possible to import a css, so it gets stored in the same location that the profiles and settings are stored.
|
||||
It should be possible to delete imported css.
|
||||
Not all css needs to be imported, linking to some path or url should be viable also - those are not deletable.
|
||||
|
||||
Settings "owns" the profiles, so changing a settings file changes the used profiles.
|
||||
At runtime, there is only 1 settings.
|
||||
It can be set using clap cli args.
|
||||
|
||||
The settings file by default is stored in a sensible user location for cli settings.
|
||||
|
||||
The Profiles are basically a dropdown.
|
||||
There is always a "Default" profile.
|
||||
The dropdown should have a an option to create a new profile - the user needs to provide a name, otherwise its a copy of the current profile.
|
||||
Profiles can be deleted; Default profile cannot be deleted.
|
||||
|
||||
Docs should be included as well.
|
||||
The navbar for docs should be collapsing accordion style for the various sections.
|
||||
There should be the following pages
|
||||
- General idea (ingestion, profiles, settings, CLI args)
|
||||
- Filtering
|
||||
- Parameters
|
||||
- Metrics
|
||||
- Dashboards
|
||||
|
||||
## FilterBar
|
||||
|
||||
Syntax aware text search.
|
||||
A parser can be written in pest.
|
||||
For now, the syntax should support regex searching and metrics searching, such as `metric_name[param]<5`.
|
||||
LogSequence ('sequence') and Timestamp ('timestamp') are reserved metrics.
|
||||
|
||||
```
|
||||
regex("error") and sequence<5 or timestamp>12:00
|
||||
```
|
||||
|
||||
DateTime parsing should support several formats
|
||||
- relative humantime (`5h`, `-3m` etc)
|
||||
- absolute
|
||||
- fulldate
|
||||
|
||||
|
||||
|
||||
## LogTimeline
|
||||
|
||||
This should work similar to how elasticsearch has a log timeline.
|
||||
Its a timeline of a barchart, where each bar is effectively some bucket of messages.
|
||||
It reflects the selected time range.
|
||||
Dragging over the log timeline displays a rectangle as it is happening and also the time range.
|
||||
Once the drag stops, the selected range becomes the new filter time range.
|
||||
|
||||
## LogPreview
|
||||
|
||||
Log preview is simply a scrollable table of the log messages, like in kibana.
|
||||
The sequence, and timestamp are visible.
|
||||
It should be possible to pick which parameters are visible.
|
||||
|
||||
## Metrics
|
||||
|
||||
Metrics can be declared from logs.
|
||||
There should be a language syntax for declaring metircs.
|
||||
|
||||
Once declared, already-ingested logs get indexed on the metric, and new metrics get ingested as well.
|
||||
|
||||
## Dashboards
|
||||
|
||||
Dashboards work the same way as in grafana.
|
||||
There should be support for promql queries to derive the visualisation.
|
||||
Supported charts should be bar chart, line chart, and pie chart.
|
||||
|
||||
|
||||
# Ideas
|
||||
|
||||
This part describes ideas that can be implemented in the future, but not now.
|
||||
|
||||
## Ntfy.sh alerts
|
||||
|
||||
Have matchers you are interested in.
|
||||
Configure the ntfy.sh channel.
|
||||
Send to the channel when an event triggers.
|
||||
|
||||
## Sequence diagram of events
|
||||
|
||||
It would be awesome if it were possible to declare participants of a sequence diagram using parameters or matchers.
|
||||
Example `node_id` for a cluster test or something.
|
||||
Then it may be possible to capture specific events of interest.
|
||||
There wouldn't be a relation between nodes, only the sequence of events determines relations in the sequence diagram.
|
||||
|
||||
## Index allocation size
|
||||
|
||||
It should be possible to see how much ram is being consumed by the program.
|
||||
What should happen if it reaches a limit?
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user