phughk/ottyl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Update README

Browse Source
This commit is contained in:
Hugh Kaznowski
2026-09-05 21:41:11 +01:00
parent 5aa7c1bae9
commit 85fb145be7
1 changed files with 36 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+36 -210
View File
@@ -1,226 +1,52 @@
# ottyl # ottyl
Observability dashboard from terminal output Observability for any command's terminal output — no logging backend, no agents, no config to ship.
# Usage
At a high level, the idea is that a program will output logs, which can be captured and displayed on a timeline and dashboard similar to grafana.
```bash ```bash
cli_command --verbose | ottyl your_app --verbose | ottyl
``` ```
# Building Pipe a program's stdout into `ottyl` and turn it into a searchable log timeline, ad-hoc metrics extracted from the log lines themselves, and Grafana-style dashboards — served from a single local binary, for the lifetime of that process:
The webapp is embedded into the `ottyl` binary at compile time, so `webapp/dist` must exist *before* running `cargo build`: - **Search:** A syntax-aware filter bar over your logs — regex matches and metric comparisons like `latency_ms[value]<200`, with relative (`5h`, `-3m`) and absolute datetimes.
- **Log timeline:** A bucketed bar chart of message volume over time, Kibana-style. Drag across it to set the active time range.
- **Metrics from logs:** Declare a metric by matching a regex against your log lines and extracting a value. It applies retroactively to everything already ingested, and to everything that follows.
- **Dashboards:** Bar, line, and pie chart panels driven by PromQL-style queries over your declared metrics.
- **Mixed sources:** Poll an existing Prometheus/OpenMetrics endpoint alongside log ingestion, in the same dashboard.
- **Profiles:** Parsing rules and dashboards are saved per app, so they're there again next time that app runs through ottyl.
## Use cases
- Debugging a noisy CLI or long-running script, instead of scrolling a terminal buffer.
- Getting metrics and charts out of a program that only ever prints text.
- Local dashboards without standing up a metrics stack.
- Combining scraped Prometheus metrics with log-derived ones during a debugging session.
## Get started
Requires Rust and Node.js. The webapp is built and embedded into the binary, so the result is a single self-contained executable.
```bash ```bash
cd webapp && npm install && npm run build && cd .. git clone <this-repo>
cargo build --release cd ottyl
make build
your_app --verbose | ottyl
``` ```
The resulting binary is self-contained — no `webapp/dist` directory or `--static-dir` flag is needed at runtime. Or point it at a log file instead of stdin, and mix in a Prometheus/OpenMetrics endpoint:
# Architecture
ottyl reads the logs via stdin and indexes it in-memory in a BTreeMap for property search.
# Engine
This is the main program, hosting the webapp over axum.
The engine reads stdin and sends line-split messages to a log ingester.
The log ingester holds a BTreeMap<IndexKey, DataValue>.
IndexKey is how we will index the logs and metrics required for search and dashboards.
```
enum IndexKey {
// Convenient way to find the start of the range
// It should never be inserted into the btreemap, but it can be used for navigating it
// This is true for all position-marker type keys
Start,
// A key for a literal raw log, where sequence is effectively the line number
// and timestamp is the time at which it was ingested into ottyl
IngestedLog{sequence: usize, timestamp: Instant},
// Convenience to get the end of the ingested log range and start of param range
LogParamSplit,
// Param declarations allow the user to specific how a parameter for a message can be derived
// This is useful for deriving timestamps from logs, that are different from the ingested timestamp (ex cat log_dump.txt)
// Other metrics may also be derived, like log level etc
ParamDeclaration{name: String},
// End of param range, start of metric declaration range
ParamMetricSplit,
// Used for storing the exact metric definition, so buckets and parameters can be derived
MetricDeclaration{name: String},
// Used to split metric declarations from their buckets
MetricBucketSplit,
//
Bucket{metric_name: String, labels: BTreeMap<String, MetricPrimitive>}
// End of key range, nothing should ever come after this
End,
}
```bash
ottyl --file /var/log/your_app.log --metrics-url http://localhost:9090/metrics
``` ```
DataValue is how we have a convenient pairing for a single BTreeMap KV mapping. ottyl prints the URL of the local web server it starts (an OS-assigned free port by default; pin one with `--port`). See `ottyl --help` for the rest, including where settings and profiles live (`--settings-path`).
It covers all the result types for the primary BTreeMap.
Full documentation — filtering syntax, parameters, metrics, dashboards — is built into the app itself, under the nav menu's docs section.
## Development
```bash
make dev # frontend dev server with hot reload, proxying to a separately running `cargo run` backend
make test # backend + frontend tests
make lint # clippy + oxlint
``` ```
enum DataValue {
// Stores the raw ingested log
// Used in IngestedLog keys
LogLiteral{text: String},
// Stores a metric declaration
// Used in MetricDeclaration keys
MetricDeclaration{message_match_regex: String, params: Vec<RegexToMetricPrimitiveMatcher>
// Stores the values of a bucket
// Used in Bucket keys
BucketValue{values: Vec<(Sequence, Timestamp, MetricPrimitive)>}
}
```
MetricPrimitive is literally just a simple way to track values without using json.
```
enum MetricPrimitive {
String(String),
Number(Number),
Timestamp(Instant),
}
```
## Tasks
The ingestion tasks reads stdin, sequences the logs, timestamps them and sends over a channel to the engine task.
The engine task processes incoming logs and incoming requests.
It should be lock-free and the sole-owner of the btreemap.
Axum is not a specific task, but exists and serves the API and frontend.
API requests get validated (like, correct syntax etc etc) and then requests are sent over a channel to the engine.
The request includes a byte buffer, for response serialisation to go into it.
The engine does the serialisation, because it is the sole owner of the btreemap and should not be copying memory simply to serialise it later.
A callback is given, so the API knows the request is processed and response is ready to send back from the byte buffer.
Rkyv can be used if that helps, otherwise sending raw bytes is fine as well.
# WebApp
ottyl provides a spa webapp dashboard in typescript react and tailwind.
It is served from an in-process axum binding.
```mermaid
block
columns 2
NavMenu
block
columns 1
FilterBar
LogTimeline
LogPreview
Metrics
Dashboards
end
```
## NavMenu
The nav menu is largely for managing settings and presets.
Presets track how to parse and index logs, and stored dashboards.
That means that restarting the application for an CLI app does not lose dashboards and parsing rules.
Settings involve colour schemes, or declaring the path for the css.
It should be possible to import a css, so it gets stored in the same location that the profiles and settings are stored.
It should be possible to delete imported css.
Not all css needs to be imported, linking to some path or url should be viable also - those are not deletable.
Settings "owns" the profiles, so changing a settings file changes the used profiles.
At runtime, there is only 1 settings.
It can be set using clap cli args.
The settings file by default is stored in a sensible user location for cli settings.
The Profiles are basically a dropdown.
There is always a "Default" profile.
The dropdown should have a an option to create a new profile - the user needs to provide a name, otherwise its a copy of the current profile.
Profiles can be deleted; Default profile cannot be deleted.
Docs should be included as well.
The navbar for docs should be collapsing accordion style for the various sections.
There should be the following pages
- General idea (ingestion, profiles, settings, CLI args)
- Filtering
- Parameters
- Metrics
- Dashboards
## FilterBar
Syntax aware text search.
A parser can be written in pest.
For now, the syntax should support regex searching and metrics searching, such as `metric_name[param]<5`.
LogSequence ('sequence') and Timestamp ('timestamp') are reserved metrics.
```
regex("error") and sequence<5 or timestamp>12:00
```
DateTime parsing should support several formats
- relative humantime (`5h`, `-3m` etc)
- absolute
- fulldate
## LogTimeline
This should work similar to how elasticsearch has a log timeline.
Its a timeline of a barchart, where each bar is effectively some bucket of messages.
It reflects the selected time range.
Dragging over the log timeline displays a rectangle as it is happening and also the time range.
Once the drag stops, the selected range becomes the new filter time range.
## LogPreview
Log preview is simply a scrollable table of the log messages, like in kibana.
The sequence, and timestamp are visible.
It should be possible to pick which parameters are visible.
## Metrics
Metrics can be declared from logs.
There should be a language syntax for declaring metircs.
Once declared, already-ingested logs get indexed on the metric, and new metrics get ingested as well.
## Dashboards
Dashboards work the same way as in grafana.
There should be support for promql queries to derive the visualisation.
Supported charts should be bar chart, line chart, and pie chart.
# Ideas
This part describes ideas that can be implemented in the future, but not now.
## Ntfy.sh alerts
Have matchers you are interested in.
Configure the ntfy.sh channel.
Send to the channel when an event triggers.
## Sequence diagram of events
It would be awesome if it were possible to declare participants of a sequence diagram using parameters or matchers.
Example `node_id` for a cluster test or something.
Then it may be possible to capture specific events of interest.
There wouldn't be a relation between nodes, only the sequence of events determines relations in the sequence diagram.
## Index allocation size
It should be possible to see how much ram is being consumed by the program.
What should happen if it reaches a limit?