RigMark
github.com/alexellis/rigmark- Category
- Developer Tools
- Rank
- No. 1599Tools index
Previous survey · No. 1547 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- GitHub
- 26 stars
- Latest release
- v0.0.4
- Date
About
RigMark is a reproducible benchmark for OpenAI-compatible local AI serving. It measures agent-shaped prose and code, a labelled structured-output ceiling, cache-busting and immediate-replay prefill, and concurrent request behaviour. It is model-agnostic and works with servers such as vLLM and SGLang. The authors state it is a serving benchmark, not a reproduction of a full multi-turn coding session.
At a glance
RigMark measures how an OpenAI-compatible inference server handles workloads resembling coding-agent traffic. It is worth using to compare serving setups under matched conditions when a peak tokens-per-second number hides slow prompt processing or contention.
Good fit
Local inference operators deciding between models, serving configurations or hardware setups.
Consider something else when
Pass if your main question is whether an agent can solve real repository tasks. The protocol checks basic output and serving behavior.
Start here
Run the same protocol against two quiet test endpoints. Compare uncached prompt processing, generation speed and concurrency results, and keep the receipts with the model and serving settings.
Cost and requirements
The MIT-licensed harness requires Python 3.10 or newer and an OpenAI-compatible endpoint with streamed output and token usage. You supply the model-serving compute.
Limits to know
Results can only isolate hardware topology when model and serving settings match. Small gains need repeated runs, and generated output still needs separate correctness evaluation.
Sources checked 2026-11-05. Source-based assessment.
Tags
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.