InferenceX
inferencex.semianalysis.com- Category
- Developer Tools
- Rank
- No. 894Tools index
Previous survey · No. 878 ·
- Listed in
- #35 Find AI benchmarks
- Pricing
- Open Source
- Type
- APP
- Builder
- semianalysisai
- GitHub
- 1.7k stars
- Latest release
- tilert-v0.1.5.post2-inferencex.1
- Date
About
An open, continuously re-run benchmark of LLM inference performance published by SemiAnalysis. Automated GitHub Actions jobs exercise inference frameworks such as vLLM, SGLang and TensorRT-LLM across NVIDIA and AMD accelerators and push over a thousand new datapoints a week to a free public dashboard covering throughput, cost efficiency and tokens per watt. Its premise is that inference software improves daily while hardware turns over annually, so point-in-time benchmarks go stale almost immediately.
What it does
InferenceX turns declared model and hardware scenarios into validated job matrices, runs them on accelerator fleets, and carries structured results through collection into a public dashboard. It also runs evaluation jobs and preserves logs, metrics, samples, and traces for diagnosis.
Why it's ranked here
InferenceX is compelling because it treats benchmark freshness as an operational pipeline, not a periodic report. Strict configuration validation, targeted smoke runs, full sweeps, artifact checks, and downstream ingestion form a credible evidence chain. The strongest value appears for teams comparing fast-moving inference stacks on supported hardware.
What's good
The repository documents clear ownership boundaries between configuration, validation, scheduling, launchers, benchmark code, collectors, and application ingestion. It distinguishes syntax checks, matrix validation, smoke runs, and full execution evidence. Result acceptance requires inspecting underlying jobs and artifacts, since a successful collector may aggregate nothing. That is unusually disciplined benchmark engineering.
Tradeoffs
Running it independently demands accelerator infrastructure, compatible fleet launchers, containers, model staging, and careful workflow operation. Local validation cannot prove allocation, server startup, network behavior, Slurm behavior, or performance. Official support covers selected NVIDIA and AMD systems, while several other platforms remain planned. The public dashboard is maintained in a separate repository.
How to use it well
Use InferenceX when maintaining inference infrastructure, selecting serving configurations, or checking whether software updates move real workload performance. Start with exact configuration generation, inspect execution-critical fields, then escalate through smoke and full sweeps. It does not replace model-quality evaluation, general hardware provisioning, or a private deployment observability system.
Technical notes+
README.md defines the project scope, Apache 2.0 licensing, dashboard, and supported hardware. docs/architecture.md describes a JSON-contract pipeline from validated declarative configuration through matrix expansion, GitHub Actions fan-out, fleet launchers, benchmark and eval artifacts, collection, and separate application ingestion. docs/testing.md separates parsing, schema checks, focused Python tests, smoke execution, trimmed sweeps, and full sweep plus eval evidence. docs/ci-procedures.md documents exact-key and filtered matrix generation, Pydantic validation, asynchronous dispatch, artifact inspection, and append-only changelog handling. docs/troubleshooting.md classifies failures by the first broken contract and warns that downstream jobs can succeed while aggregating empty upstream output.
Observed
- License
- Apache 2.0
- Public interface
- Free public performance dashboard
- Execution interfaces
- Python validation and generation, Bash launchers, GitHub Actions workflows
- Supported accelerator families
- NVIDIA GB300, GB200, B300, B200, H200, H100 and AMD MI355X, MI325X, MI300X
- Documentation languages
- English documentation with paired Chinese pages
- Repository structure
- Configuration, validation, workflow orchestration, fleet launchers, benchmark execution, artifact collection, and downstream ingestion are separate layers
Read from README.md, docs/index.md, docs/testing.md, docs/index_zh.md, docs/procedures.md, docs/testing_zh.md, docs/agent-guide.md, docs/architecture.md, docs/ci-procedures.md, docs/procedures_zh.md, docs/agent-guide_zh.md, docs/architecture_zh.md, docs/troubleshooting.md.
Intel on InferenceX
Tags
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.