diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ea8c4bf --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +/target diff --git a/Cargo.toml b/Cargo.toml new file mode 100644 index 0000000..50c58c5 --- /dev/null +++ b/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "ottyl" +version = "0.1.0" +edition = "2024" + +[dependencies] diff --git a/README.md b/README.md index f97a49d..5443714 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,177 @@ # ottyl -Observability dashboard from terminal output \ No newline at end of file +Observability dashboard from terminal output + +# Usage + +At a high level, the idea is that a program will output logs, which can be captured and displayed on a timeline and dashboard similar to grafana. + +```bash +cli_command --verbose | ottyl + +``` + +# Architecture + +ottyl reads the logs via stdin and indexes it in-memory in a BTreeMap for property search. + +# Engine + +This is the main program, hosting the webapp over axum. + +The engine reads stdin and sends line-split messages to a log ingester. +The log ingester holds a mutex over a BTreeMap. + +The BTreeMap may be a custom variation of the std library one, because we need to be able to scan specific ranges and jump to keys. +``` +/// Get the given key, or the next key in the btree +fn get_or_next(&self, key: &K) -> Option>{...} + +/// Get the next key in the btree +fn get_or_next(&self, key: &K) -> Option>{...} + +/// Get the previous key in the btree +fn previous(&self, key: &K) -> Option> {...} +``` + +IndexKey is how we will index the logs and metrics required for search and dashboards. +``` +enum IndexKey { + // Convenient way to find the start of the range + // It should never be inserted into the btreemap, but it can be used for navigating it + // This is true for all position-marker type keys + Start, + // A key for a literal raw log, where sequence is effectively the line number + // and timestamp is the time at which it was ingested into ottyl + IngestedLog{sequence: usize, timestamp: Instant}, + // Convenience to get the end of the ingested log range and start of param range + LogParamSplit, + // Param declarations allow the user to specific how a parameter for a message can be derived + // This is useful for deriving timestamps from logs, that are different from the ingested timestamp (ex cat log_dump.txt) + // Other metrics may also be derived, like log level etc + ParamDeclaration{name: String}, + // End of param range, start of metric declaration range + ParamMetricSplit, + // Used for storing the exact metric definition, so buckets and parameters can be derived + MetricDeclaration{name: String}, + // Used to split metric declarations from their buckets + MetricBucketSplit, + // + Bucket{metric_name: String, labels: BTreeMap} + // End of key range, nothing should ever come after this + End, +} + +``` + +DataValue is how we have a convenient pairing for a single BTreeMap KV mapping. +It covers all the result types for the primary BTreeMap. + +``` +enum DataValue { + // Stores the raw ingested log + // Used in IngestedLog keys + LogLiteral{text: String}, + // Stores a metric declaration + // Used in MetricDeclaration keys + MetricDeclaration{message_match_regex: String, params: Vec + // Stores the values of a bucket + // Used in Bucket keys + BucketValue{values: Vec<(Sequence, Timestamp, MetricPrimitive)>} +} +``` + +MetricPrimitive is literally just a simple way to track values without using json. +``` +enum MetricPrimitive { + String(String), + Number(Number), + Timestamp(Instant), +} +``` + + +# WebApp + +ottyl provides a spa webapp dashboard in typescript react and tailwind. +It is served from an in-process axum binding. + +```mermaid +block +columns 2 + NavMenu + block + columns 1 + FilterBar + LogTimeline + LogPreview + Metrics + Dashboards + end +``` + +## NavMenu + +The nav menu is largely for managing settings and presets. + +Presets track how to parse and index logs, and stored dashboards. +That means that restarting the application for an CLI app does not lose dashboards and parsing rules. + +Settings involve colour schemes, or declaring the path for the css. +It should be possible to import a css, so it gets stored in the same location that the profiles and settings are stored. +It should be possible to delete imported css. +Not all css needs to be imported, linking to some path or url should be viable also - those are not deletable. + +Settings "owns" the profiles, so changing a settings file changes the used profiles. +At runtime, there is only 1 settings. +It can be set using clap cli args. + +The settings file by default is stored in a sensible user location for cli settings. + +## FilterBar + +Syntax aware text search. +A parser can be written in pest. +For now, the syntax should support regex searching and metrics searching, such as `metric_name[param]<5`. +LogSequence ('sequence') and Timestamp ('timestamp') are reserved metrics. + +``` +regex("error") and sequence<5 or timestamp>12:00 +``` + +DateTime parsing should support several formats +- relative humantime (`5h`, `-3m` etc) +- absolute +- fulldate + + + +## LogTimeline + +This should work similar to how elasticsearch has a log timeline. +Its a timeline of a barchart, where each bar is effectively some bucket of messages. +It reflects the selected time range. +Dragging over the log timeline displays a rectangle as it is happening and also the time range. +Once the drag stops, the selected range becomes the new filter time range. + +## LogPreview + +Log preview is simply a scrollable table of the log messages, like in kibana. +The sequence, and timestamp are visible. +It should be possible to pick which parameters are visible. + +## Metrics + +Metrics can be declared from logs. +There should be a language syntax for declaring metircs. + +Once declared, already-ingested logs get indexed on the metric, and new metrics get ingested as well. + +## Dashboards + +Dashboards work the same way as in grafana. +There should be support for promql queries to derive the visualisation. +Supported charts should be bar chart, line chart, and pie chart. + + + diff --git a/src/main.rs b/src/main.rs new file mode 100644 index 0000000..e7a11a9 --- /dev/null +++ b/src/main.rs @@ -0,0 +1,3 @@ +fn main() { + println!("Hello, world!"); +}