ctrlb-decompose
github.com/ctrlb-hq/ctrlb-decompose- Category
- Developer Tools
- Rank
- No. 1149Tools index
Previous survey · No. 1130 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Data Processing
- Interfaces
- CLI
- GitHub
- 305 stars
- Latest release
- v0.1.0
- Date
About
A Rust-based log analysis tool that clusters millions of noisy log lines into a small set of typed, statistically-annotated patterns using CLP encoding and Drain3 clustering. It runs fully locally (CLI, Rust library, or WASM in-browser) and outputs human-readable, JSON, or compact LLM-optimized formats so large log files can be summarized for debugging or fed into an LLM context window.
What it does
Point it at a log file or a pipe and it hands back a ranked list of line shapes instead of the lines themselves. Each line loses its timestamp, then numbers, IP addresses and IDs are swapped for typed placeholders, so lines that differ only in values look identical. A prefix-tree clusterer groups near-identical shapes and turns the words that vary into wildcards. For every group it keeps running statistics: percentiles for numeric fields, estimates of unique values, a few sample lines. Finally it flags odd behaviour, such as a burst late in the file, and ranks groups by volume weighted by words like ERROR or TIMEOUT.
Why it's ranked here
Log clustering is old research. The value here is packaging it for the place people now read logs: a model's context window. In the project's own demo a 56,482-line Apache error log collapses to 25 patterns, and the compact output is built for pasting into a prompt, with long tokens trimmed and two sample lines per pattern by default. It ships as a command-line tool, a Rust library and a browser build, MIT licensed, with install paths for macOS, Debian, Windows and Docker, plus a Claude Code skill. The caveat is maturity: the manifest declares version 0.1.0, so treat it as a sharp new tool, not settled infrastructure.
What's good
Memory stays bounded: at most 10,000 clusters with the least recently used evicted, fixed-size sketches instead of stored numbers, and sampled example lines, so a large file streams through in one pass. Severity detection is careful about false alarms: it reads only the first 100 characters of each pattern and splits on punctuation, so a JSON field named error deep in a line does not mark it as a failure, and a unit test pins that case. The compact output also cuts any token longer than 50 characters down to its head and tail, which strips auth tokens and base64 blobs before they eat context.
Tradeoffs
Some heuristics are blunter than their names. The bimodal flag fires whenever the 75th percentile is more than five times the 25th, which measures spread, not two peaks. Severity is keyword matching, so a log that encodes its level as a number never ranks as an error. The compact output prints times without a date, so a log spanning midnight shows a range that looks backwards, as the project's own demo does. That mode never prints the anomaly flags, which only reorder error and warning patterns, yet the bundled agent skill tells the model to report anomalies. The command line also prints a CtrlB advert unless run quietly.
How to use it well
Use it as the first pass on an unfamiliar log before a person or an agent reads anything raw: run the compact mode with a cap of twenty patterns and paste the result, or install the skill so a coding agent does it for you. It fits incident triage on one file or one stream. It is not a log store: nothing is indexed or kept between runs, so it does not replace search or retention, and comparing two windows means two runs. When any error or warning pattern exists, the compact mode lists only five other patterns, so switch to JSON when you need the full inventory.
Technical notes+
src/lib.rs runs one pass per line: extract_timestamp and strip_timestamp, then ClpDrainPipeline::process_line, then PatternStore::accumulate; after input ends, finalize, detect_anomalies and compute_scores run once. The CLI reads with read_until plus from_utf8_lossy, so invalid UTF-8 is replaced rather than rejected, while the WASM entry process_log_text takes the whole input as one &str. src/extraction/pipeline.rs builds EncodingContext::new(2048, 128) and merges CLP placeholders with Drain wildcards through two cursor pairs. src/extraction/drain3.rs defaults to log_cluster_depth 4, sim_th 0.4, max_children 100 and max_clusters 10000 on an LruCache, and panics on a depth below 3. src/scoring.rs computes count * keyword_weight * (1 + max anomaly severity) with weights 10, 5 and 0.1, scanning only KEYWORD_PREFIX_LEN = 100 characters. src/anomaly.rs thresholds: LowCardinality at count > 100 with under 5 uniques, ClusteredNumeric when (max - min) divided by mean is under 0.05, BimodalDistribution when p75 exceeds five times p25, FrequencySpike on a 3x rate ratio in the first or last 10% of time buckets. src/format/llm.rs routes Error and Warn severities to a critical list, caps the rest at high_volume_limit = 5, never renders AnomalyFlags, and prints the header span with %H:%M:%S only; to_format_options sets context to 2 under --llm. tests/integration.rs shells out to cargo run against a sample fixture.
Observed
- License
- MIT (LICENSE file, declared in Cargo.toml)
- Language
- Rust, edition 2024, minimum toolchain 1.94.0
- Interfaces
- CLI (default feature), Rust library, WebAssembly build behind a wasm feature flag
- Output formats
- Colored terminal text, compact markdown for LLM prompts, structured JSON
- Install surface
- Homebrew tap, Debian package, Windows x64 ZIP, Debian and Alpine Dockerfiles, cargo build from source
- Agent integration
- Claude Code skill in plugin/skills/analyze-logs/SKILL.md
- Tests
- Inline unit tests plus CLI integration tests in tests/integration.rs
- Clustering defaults
- Drain similarity threshold 0.4, LRU cap of 10,000 clusters (src/extraction/drain3.rs)
- Error stream output
- CLI prints a CtrlB promotional line to stderr unless -q is passed (src/lib.rs)
Read from README.md, Cargo.toml, LICENSE, src/main.rs, src/lib.rs, src/extraction/pipeline.rs, src/extraction/drain3.rs, src/anomaly.rs, src/scoring.rs, src/format/llm.rs, plugin/skills/analyze-logs/SKILL.md, tests/integration.rs.
What it can do
Cluster millions of noisy log lines into a small set of typed patterns
Raw log files → Typed log patterns
Annotate log patterns with statistical quantiles
Clustered log patterns → Quantile statistics
Detect anomalies in log data
Log lines → Anomaly annotations
Output analysis results in human-readable format
Log patterns → Human-readable text
Output analysis results in JSON format
Log patterns → JSON data
Output a compact LLM-optimized summary format
Log patterns → LLM-optimized text
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.