MLX Benchmark
github.com/goekdeniz-guelmez/mlx-benchmark- Category
- Developer Tools
- Rank
- No. 1959Tools index
Previous survey · No. 1865 ·
- Listed in
- #53 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Builder
- @ActuallyIsaak
- GitHub
- 38 stars
- Latest release
- v1.1.0
- Date
About
A benchmark harness that scores how well language models know Apple's MLX framework and can write MLX code. It ships a 441-question dataset spanning knowledge, multiple-choice, true/false, code completion, full coding tasks and debugging across four difficulty levels and eleven categories, and runs models locally through Ollama or against the Anthropic, OpenAI, Groq and OpenRouter APIs. Results are saved as JSON with per-question scores and can be rendered into comparison charts and LaTeX tables.
What it does
It sends a fixed set of MLX questions to a chosen model, then checks structured answers with simple matching and open responses with another model. Runs can target selected topics, task forms, or difficulty bands. Results preserve each answer and summarize accuracy for later comparison.
Why it's ranked here
This is a focused, reproducible harness with useful provider choice, filtering, saved evidence, and publication-oriented exports. The verdict is tempered by its evaluation design: most responses receive a binary decision from a model judge, which defaults to the model being tested unless configured otherwise.
What's good
Local Ollama support keeps experiments on your machine, while optional cloud backends make cross-provider comparisons practical. Deterministic default sampling, configurable judges, concurrent workers, custom datasets, and per-question records support controlled runs. Structured questions avoid unnecessary judge calls through direct answer matching.
Tradeoffs
Open-ended scoring depends on a model judge and accepts only correct or incorrect, with no partial credit. Failed generation calls are omitted from recorded totals, which can make completion reliability less visible in aggregate accuracy. The PNG chart excludes the very-hard difficulty band, although the LaTeX output includes it.
How to use it well
Use it when selecting or regression-testing models for MLX documentation, code generation, and debugging work. Start with a limited filtered run, assign a stronger independent judge, then inspect saved answers before trusting aggregate comparisons. It does not measure general model quality, execution correctness, latency, cost, or production reliability.
Technical notes+
pyproject.toml defines a Python 3.10+ setuptools package, the mlx-bench CLI entry point, bundled JSONL package data, and optional provider, plotting, configuration, and development dependencies. mlx_benchmark/runner.py coordinates generation and judging, supports ThreadPoolExecutor workers, aggregates BenchStats, and writes JSON results. mlx_benchmark/judge.py uses quick_match for true/false and multiple-choice items, then llm_judge for semantic evaluation. mlx_benchmark/export.py creates booktabs LaTeX tables and matplotlib PNG charts. mlx_benchmark/dataset.py loads custom or bundled JSONL data, while mlx_benchmark/backends/ supplies Ollama, Anthropic, and OpenAI-compatible adapters.
Observed
- License
- MIT
- Primary language
- Python
- Python requirement
- Python 3.10 or newer
- Installation
- Published as the pip package mlx-benchmark with optional extras
- Interfaces
- Command-line interface and importable Python API
- Providers
- Ollama, Anthropic, OpenAI, Groq, and OpenRouter
- Packaging
- Setuptools build with bundled JSONL package data
Read from README.md, pyproject.toml, requirements.txt, mlx_benchmark/cli.py, mlx_benchmark/judge.py, mlx_benchmark/export.py, mlx_benchmark/runner.py, mlx_benchmark/dataset.py, mlx_benchmark/prompts.py, mlx_benchmark/version.py, mlx_benchmark/__init__.py, mlx_benchmark/backends/base.py, mlx_benchmark/backends/__init__.py, mlx_benchmark/backends/ollama_backend.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.