Sprocket
sprocket.bio- Category
- Developer Tools
- Rank
- No. 1144Tools index
Previous survey · No. 1167 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Workflow Automation
- Interfaces
- CLI
- GitHub
- 221 stars
- Latest release
- v0.33.0
- Date
About
Sprocket is an open-source, Rust-built workflow engine for running Workflow Description Language (WDL) bioinformatics pipelines, letting researchers write, lint, and run workflows locally with a Docker backend before scaling the same code to HPC clusters (Slurm, LSF) or cloud storage (S3, Azure, GCS) via a Task Execution Service. Built-in linting and a VS Code/Neovim language server catch WDL errors before a run starts, and the project targets 20,000+ concurrent jobs at scale.
What it does
Sprocket is one command-line program for the whole life of a WDL file, the workflow language many genomics groups use to describe chained analysis steps. It parses and type-checks documents, applies 38 style and correctness rules, reformats code, writes a template of the inputs a task expects, checks a filled-in input file before anything runs, and then executes the task or workflow, with Docker as a configurable container backend. The same parser also powers an editor language server and a Python package for reading WDL from scripts.
Why it's ranked here
The argument is breadth from one codebase, with real discipline behind it. Checker, formatter, linter, language server and execution engine live in a single Rust workspace of sixteen member crates, so the analysis an editor shows is the analysis the command line runs. It is dual licensed MIT or Apache 2.0 and ships through Cargo, Homebrew, a container image and a Nix flake. Its CI replays analysis against a list of outside WDL repositories so any change in diagnostics gets reviewed. The main reservation: the run server and WDL test runner are still labelled experimental.
What's good
The lint rules target how pipelines actually break: container images on mutable tags, hard-coded host paths, software installed inline inside a command, glob-star patterns and missing recommended runtime keys, plus one rule that runs ShellCheck over task command scripts. Any rule can be switched off in configuration or silenced for a single block with a comment directive. An experimental lock command pins Docker images to a sha256 digest, so a rerun pulls the same image. Input validation catches missing, mistyped, extra and nonexistent file inputs before a job is submitted.
Tradeoffs
The headline scale figure, 20,000 or more concurrent jobs, appears as a guiding target in the project's own principles, not as a measured result. The HTTP run server, WDL unit tests and documentation generator all sit under a subcommand labelled developmental and experimental. Stopping Sprocket mid-run leaves running Docker containers behind, and files they wrote can end up owned by root. The project warns its prebuilt Linux binary may not work on every distribution, and the Python package exposes parsing, syntax tree and diagnostics modules only, with no execution.
How to use it well
Best fit is a team already writing WDL that wants editor feedback, consistent formatting and pre-flight input checks as part of code review. Put the lint and format checks in CI, generate input templates instead of hand-writing them, and validate inputs before every submission. Local runs through Docker are the clearly supported path; confirm cluster and cloud backend support against current documentation before committing a production pipeline to it. It reads only WDL, so pipelines written in other workflow languages get nothing from it.
Technical notes+
src/main.rs installs mimalloc as the global allocator, citing the allocation-heavy multithreaded parse and analysis workload, then calls sprocket::sprocket_main. src/commands.rs defines the Commands enum (Analyzer, Check, Completions, Config, Explain, Format, Inputs, Lint, Run, Validate) and DevCommands (Doc, Lock, Module, Server, Test); uses_docker_backend and warn_docker_termination document the orphaned-container behaviour. crates/wdl-lint/src/lib.rs registers 38 rules in all_rules, drops RuleSeverity::Off rules in rules(), and in debug builds panics on non-PascalCase, duplicate, unconfigured or self-referencing rule IDs. crates/wdl-engine/src/lib.rs wraps an LRU cache of tokio OnceCell entries with CancellationToken-aware initialisation, hashes content with blake3, and maps file and directory inputs onto crankshaft input types. Cargo.toml pulls crankshaft with the docker feature, tes with oauth, cloud-copy, axum, utoipa and sqlx on SQLite, and sets rust-version 1.95 on edition 2024. python/README.md lists the sprocket_bio grammar, ast and diagnostics modules. CONTRIBUTING.md describes the gauntlet crate replaying analysis against external WDL repositories pinned in Gauntlet.toml.
Observed
- License
- MIT OR Apache-2.0, at the user's choice
- Language
- Rust, edition 2024, minimum supported Rust 1.95
- Install surface
- cargo install, Homebrew (macOS and Linux), container image on GHCR, Nix flake, prebuilt release binaries
- Interfaces
- CLI; language server via sprocket analyzer; experimental HTTP API server; Rust library crates; Python package sprocket-bio
- Lint rules
- 38 rules registered in crates/wdl-lint/src/lib.rs, each switchable by severity or comment directive
- Python bindings
- Python 3.10 or newer; grammar, ast and diagnostics modules, no execution
- Execution backends in manifest
- Docker via crankshaft; Task Execution Service client (tes crate)
- Regression harness
- Gauntlet replays analysis against external WDL repositories (CONTRIBUTING.md)
- AI contribution policy
- AI assistance must be disclosed; commits authored by an LLM agent are not accepted
Read from README.md, Cargo.toml, crates/README.md, src/main.rs, src/commands.rs, crates/wdl-engine/src/lib.rs, crates/wdl-lint/src/lib.rs, crates/wdl-lsp/src/lib.rs, python/README.md, CONTRIBUTING.md.
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.