Intel / article

Evals Skills for Coding Agents

Source: hamel.dev
Author: Hamel Husain
Date: Published Mar 2, 2026

Context

Access to an platform does not tell a coding what constitutes a product failure. Hamel Husain's evals-skills release supplies reusable instructions for examining interaction logs, defining failure categories and checking automated evaluators against human labels.

His example separates a support bot misstating a policy from falsely claiming it performed an action for the user. Combining those problems into one generic score can hide which behavior needs fixing. The skills provide a starting method for product-specific evaluation, which teams then adapt to their own data and requirements. His evaluation guide explains the underlying human- process.

Key takeaways · AI-distilled

OpenAI built a product with Codex agents — three engineers, five months, roughly a million lines of code — and found that improving the infrastructure around the agent mattered more than improving the model. The agents queried traces to check their own work.
Three layers do different jobs for an agent: documentation tells it what to do, telemetry tells it whether the thing ran, and evals tell it whether the output was any good.
A judge prompt should force a binary Pass/Fail, and the judge itself needs validating — calibrate it against human labels using true-positive and true-negative rates, then correct for its bias.
has to be scored in two separate pieces: did retrieval fetch the right documents, and did generation use them well. A single combined number hides which half is broken.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.

Key quotes

“They built a product entirely with Codex agents — three engineers, five months, ~1 million lines of code — and found that improving the infrastructure around the agent mattered more than improving the model.”

“Documentation tells the agent what to do. Telemetry tells it whether it worked. Evals tell it whether the output is good.”

“If you lump them together in a generic “hallucination score,” you’ll miss errors.”

“These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them.”

Read the source hamel.dev