Vibeleaderboard
← All Intel
Intel / video

Why Building an Eval Platform Is Harder Than It Looks — Braintrust

Source
youtube.com
Author
AI Engineer
Date

Context

Hossein Niazmandi, who leads solutions engineering in the West at Braintrust, argues in an AI Engineer talk that measuring an 's quality takes more than putting a screen on a spreadsheet. Braintrust sells software, so the talk is also a product case. He says teams need both evals before release, which test an agent against a set of cases, and observability after, which records what it does in production, because language models give different answers to the same input.

He lays out the stages teams pass through: a spreadsheet, a custom interface, side-by-side experiments that product managers and domain experts can run, and finally a loop where failures seen in production become new test cases. At that point, he says, the hard part is data, because agent traces are large nested JSON records that need both real-time and long-running queries, which is why Braintrust built its own query language, BTQL. The summary lists no measurements, so the stages are his account of a common path.

Key takeaways · AI-distilled
  • Braintrust's Niazmandi frames agent quality as two pillars, evals before production and observability after, and argues non-deterministic LLMs make both necessary.
  • He describes four maturity stages: spreadsheets, a custom UI, side-by-side experiments for PMs and domain experts, and a flywheel that turns production failures into test cases.
  • At the flywheel stage the hard problem becomes data: agent traces are huge nested JSON needing both real-time and long-running queries, which he says break typical data warehouses and led Braintrust to build BTQL.
  • The talk ends with coding agents running evals and surfacing failures nobody thought to look for, while humans review results and decide.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…