Auto-Inference-Optimiser
github.com/manthanguptaa/auto-inference-optimiser- Category
- AI Agents
- Rank
- No. 1176Tools index
Previous survey · No. 1158 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- manthanguptaa
- GitHub
- 38 stars
- Date
About
An autonomous AI agent that automatically optimizes LLM inference speed on Apple Silicon by running experiments overnight. It hill-climbs on tokens per second by modifying inference code and uses git commits as experiment tracking.
What it does
It provides a controlled laboratory for testing changes to an MLX inference pipeline. A fixed harness measures decoding speed, prompt processing, memory use, perplexity, and five task checks. Candidate changes survive only when generation throughput improves and both quality gates pass.
Why it's ranked here
The project makes performance experimentation unusually inspectable. Published M4 runs show a 10.7% decoding gain for a 0.5B Qwen model and 2.2% for a 3B Llama model. More importantly, its task checks rejected quantization changes that perplexity alone would have accepted. The evidence remains narrow, but the method is concrete and reproducible.
What's good
The locked benchmark separates measurement code from the editable inference pipeline. Warmup runs reduce Metal compilation distortion, repeated runs are averaged, and failed experiments are reverted. The change monitor also flags suspect shortcuts, including reduced generation length. Results report prompt speed, memory, perplexity, and task correctness alongside the main decoding metric.
Tradeoffs
The acceptance rule favors decoding throughput, so prompt speed and output behavior can worsen while an experiment survives. Greedy decoding produced the largest gains but removed sampling diversity and reduced one model's task score. The five content checks are simple keyword and pattern tests, not broad quality evaluation. Published measurements cover one M4 machine, two model families, and small quantized models.
How to use it well
Use it when tuning MLX inference on an Apple Silicon Mac and you can supervise agent-written experiments through Git. It suits repeatable local optimization work where decoding throughput is the main goal. It does not provide production serving, a network API, non-Apple acceleration, or comprehensive model-quality assessment.
Technical notes+
prepare.py fixes BENCHMARK_MODEL, MAX_TOKENS, three measured runs, one warmup, a perplexity threshold of 50.0, and a sanity threshold of 0.6. It imports generate_text behavior from the editable inference.py, computes aggregate performance and quality metrics, and reports configuration changes against the first Git revision. inference.py uses MLX stream_generate with a singleton argmax sampler and a prefill step size of 2048. program.md directs an external coding agent to modify only inference.py, commit each experiment, run prepare.py, append results to results.tsv, and reset failed or slower commits. requirements.txt lists MLX, MLX-LM, NumPy, Transformers, and Hugging Face Hub dependencies.
Observed
- Primary language
- Python
- Installation
- Dependencies install through pip from requirements.txt.
- Interface
- Command-line benchmark and external coding-agent workflow; no network API is described.
- Platform support
- macOS on Apple Silicon M1, M2, M3, or M4, with Python 3.10 or newer.
- Inference stack
- MLX and MLX-LM drive local model loading, generation, and Metal memory measurements.
- Repository structure
- The supplied project consists of a benchmark harness, one editable inference pipeline, an agent protocol, and a dependency list.
Read from README.md, requirements.txt, prepare.py, inference.py, program.md.
What it can do
Optimize LLM inference speed on Apple Silicon
LLM inference code → Optimized inference code with improved tokens per second
Run automated optimization experiments overnight
Base inference implementation → Series of experimental code modifications
Track optimization experiments via git commits
Code changes from experiments → Git repository with commit history of experiments
Perform hill-climbing optimization on tokens per second
Performance metrics from inference runs → Iteratively improved code configurations
Modify inference code automatically
Current inference implementation → Modified code variants for testing
Measure and benchmark LLM inference performance
Modified inference code → Tokens per second performance metrics
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.