Vibeleaderboard
Index / tool
Visit github.com
Category
Developer Tools
Rank
No. 1599Tools index

Previous survey · No. 1547 ·

Pricing
Open Source
Type
TOOL
Use case
Model & Agent Evaluation
GitHub
26 stars
Latest release
v0.0.4
Date

About

RigMark is a reproducible benchmark for OpenAI-compatible local AI serving. It measures agent-shaped prose and code, a labelled structured-output ceiling, cache-busting and immediate-replay prefill, and concurrent request behaviour. It is model-agnostic and works with servers such as vLLM and SGLang. The authors state it is a serving benchmark, not a reproduction of a full multi-turn coding session.

At a glance

RigMark measures how an OpenAI-compatible inference server handles workloads resembling coding-agent traffic. It is worth using to compare serving setups under matched conditions when a peak tokens-per-second number hides slow prompt processing or contention.

Good fit

Local inference operators deciding between models, serving configurations or hardware setups.

Consider something else when

Pass if your main question is whether an agent can solve real repository tasks. The protocol checks basic output and serving behavior.

Start here

Run the same protocol against two quiet test endpoints. Compare uncached prompt processing, generation speed and concurrency results, and keep the receipts with the model and serving settings.

Cost and requirements

The MIT-licensed harness requires Python 3.10 or newer and an OpenAI-compatible endpoint with streamed output and token usage. You supply the model-serving compute.

Limits to know

Results can only isolate hardware topology when model and serving settings match. Small gains need repeated runs, and generated output still needs separate correctness evaluation.

Tags

benchmarklocal-aillmvllmsglangcoding-agent

Media

RigMark

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.