Initial readme/spec

README.md

@@ -1,3 +1,177 @@
 # ottyl
 
-Observability dashboard from terminal output
+Observability dashboard from terminal output
+
+# Usage
+
+At a high level, the idea is that a program will output logs, which can be captured and displayed on a timeline and dashboard similar to grafana.
+
+```bash
+cli_command --verbose | ottyl
+
+```
+
+# Architecture
+
+ottyl reads the logs via stdin and indexes it in-memory in a BTreeMap for property search.
+
+# Engine
+
+This is the main program, hosting the webapp over axum.
+
+The engine reads stdin and sends line-split messages to a log ingester.
+The log ingester holds a mutex over a BTreeMap<IndexKey, DataValue>.
+
+The BTreeMap may be a custom variation of the std library one, because we need to be able to scan specific ranges and jump to keys.
+```
+/// Get the given key, or the next key in the btree
+fn get_or_next(&self, key: &K) -> Option<Entry<K, V>>{...} 
+
+/// Get the next key in the btree
+fn get_or_next(&self, key: &K) -> Option<Entry<K, V>>{...} 
+
+/// Get the previous key in the btree
+fn previous(&self, key: &K) -> Option<Entry<K, V>> {...}
+```
+
+IndexKey is how we will index the logs and metrics required for search and dashboards.
+```
+enum IndexKey {
+  // Convenient way to find the start of the range
+  // It should never be inserted into the btreemap, but it can be used for navigating it
+  // This is true for all position-marker type keys
+  Start,
+  // A key for a literal raw log, where sequence is effectively the line number
+  // and timestamp is the time at which it was ingested into ottyl
+  IngestedLog{sequence: usize, timestamp: Instant},
+  // Convenience to get the end of the ingested log range and start of param range
+  LogParamSplit,
+  // Param declarations allow the user to specific how a parameter for a message can be derived
+  // This is useful for deriving timestamps from logs, that are different from the ingested timestamp (ex cat log_dump.txt)
+  // Other metrics may also be derived, like log level etc
+  ParamDeclaration{name: String},
+  // End of param range, start of metric declaration range
+  ParamMetricSplit,
+  // Used for storing the exact metric definition, so buckets and parameters can be derived
+  MetricDeclaration{name: String},
+  // Used to split metric declarations from their buckets
+  MetricBucketSplit,
+  // 
+  Bucket{metric_name: String, labels: BTreeMap<String, MetricPrimitive>}
+  // End of key range, nothing should ever come after this
+  End,
+}
+
+```
+
+DataValue is how we have a convenient pairing for a single BTreeMap KV mapping.
+It covers all the result types for the primary BTreeMap.
+
+```
+enum DataValue {
+  // Stores the raw ingested log
+  // Used in IngestedLog keys
+  LogLiteral{text: String},
+  // Stores a metric declaration
+  // Used in MetricDeclaration keys
+  MetricDeclaration{message_match_regex: String, params: Vec<RegexToMetricPrimitiveMatcher>
+  // Stores the values of a bucket
+  // Used in Bucket keys
+  BucketValue{values: Vec<(Sequence, Timestamp, MetricPrimitive)>}
+}
+```
+
+MetricPrimitive is literally just a simple way to track values without using json.
+```
+enum MetricPrimitive {
+  String(String),
+  Number(Number),
+  Timestamp(Instant),
+}
+```
+
+
+# WebApp
+
+ottyl provides a spa webapp dashboard in typescript react and tailwind.
+It is served from an in-process axum binding.
+
+```mermaid
+block
+columns 2
+  NavMenu
+  block
+  columns 1
+    FilterBar
+    LogTimeline
+    LogPreview
+    Metrics
+    Dashboards
+  end
+```
+
+## NavMenu
+
+The nav menu is largely for managing settings and presets.
+
+Presets track how to parse and index logs, and stored dashboards.
+That means that restarting the application for an CLI app does not lose dashboards and parsing rules.
+
+Settings involve colour schemes, or declaring the path for the css.
+It should be possible to import a css, so it gets stored in the same location that the profiles and settings are stored.
+It should be possible to delete imported css.
+Not all css needs to be imported, linking to some path or url should be viable also - those are not deletable.
+
+Settings "owns" the profiles, so changing a settings file changes the used profiles.
+At runtime, there is only 1 settings.
+It can be set using clap cli args.
+
+The settings file by default is stored in a sensible user location for cli settings.
+
+## FilterBar
+
+Syntax aware text search.
+A parser can be written in pest.
+For now, the syntax should support regex searching and metrics searching, such as `metric_name[param]<5`.
+LogSequence ('sequence') and Timestamp ('timestamp') are reserved metrics.
+
+```
+regex("error") and sequence<5 or timestamp>12:00
+```
+
+DateTime parsing should support several formats
+- relative humantime (`5h`, `-3m` etc)
+- absolute
+- fulldate
+
+
+
+## LogTimeline
+
+This should work similar to how elasticsearch has a log timeline.
+Its a timeline of a barchart, where each bar is effectively some bucket of messages.
+It reflects the selected time range.
+Dragging over the log timeline displays a rectangle as it is happening and also the time range.
+Once the drag stops, the selected range becomes the new filter time range.
+
+## LogPreview
+
+Log preview is simply a scrollable table of the log messages, like in kibana.
+The sequence, and timestamp are visible.
+It should be possible to pick which parameters are visible.
+
+## Metrics
+
+Metrics can be declared from logs.
+There should be a language syntax for declaring metircs.
+
+Once declared, already-ingested logs get indexed on the metric, and new metrics get ingested as well.
+
+## Dashboards
+
+Dashboards work the same way as in grafana.
+There should be support for promql queries to derive the visualisation.
+Supported charts should be bar chart, line chart, and pie chart.
+
+
+