OpenAI built a product with Codex agents — three engineers, five months, roughly a million lines of code — and found that improving the infrastructure around the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → mattered more than improving the model. The agents queried traces to check their own work.
Three layers do different jobs for an agent: documentation tells it what to do, telemetry tells it whether the thing ran, and evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → tell it whether the output was any good.
A judge prompt should force a binary Pass/Fail, and the judge itself needs validating — calibrate it against human labels using true-positive and true-negative rates, then correct for its bias.
RAGRetrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.Full definition → has to be scored in two separate pieces: did retrieval fetch the right documents, and did generation use them well. A single combined number hides which half is broken.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
Why it matters
If your coding agent is instrumenting and evaluating your AI product, these skills encode the error-analysis discipline that keeps it from lumping distinct failure modes into one useless score.
Key quotes
“They built a product entirely with Codex agents — three engineers, five months, ~1 million lines of code — and found that improving the infrastructure around the agent mattered more than improving the model.”
“Documentation tells the agent what to do. Telemetry tells it whether it worked. Evals tell it whether the output is good.”
“If you lump them together in a generic “hallucination score,” you’ll miss errors.”
“These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them.”
