From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Shows how to make agent benchmark scores reproducible and comparable by recording every configuration layer and running repeats, so results reflect reliability instead of luck.
Key takeaways · AI-distilled
Weights & Biases' WolfBench reports average, best and worst runs plus tasks solved at least once versus every time, because repeating a configuration separates a lucky success from dependable behavior.
Wolfram Ravenwolf treats a result as a six-layer configuration: benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →, runner, sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition →, AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →, model service and settings. Versions, provider routing, system prompts, resource limits and timeouts all change what the score measures.
Two traps: a cheaper model can burn enough tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → to raise total cost, and higher reasoning effort can lower the score by leaving too little time to run commands. Infrastructure failures need separate handling, and zero temperature does not guarantee identical runs.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
videoHow Secrets Leak Through AI Agents (and How to Stop It) — Venice AI
videoBuild a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
videoContinuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
videoLet Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
