Vibeleaderboard
← All Intel
Intel / video

Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

Source
youtube.com
Author
AI Engineer
Date

Context

Browserbase's Miguel González Fernández and Microsoft Research's Corby Rosset argue that the automated judges bundled with web- benchmarks overstate how often agents succeed. A judge here is a model that decides whether an agent completed its task. In their talk, an official judge scored an agent as succeeding 74 percent of the time while their better verifier put it at 38 percent. Training an agent against a judge that approves failures, they say, produces a more confident agent rather than a better one.

Their verifier, the Universal Verifier, grades against task-specific rubrics and ranks the most relevant screenshots as evidence for each criterion. The accompanying paper reports false positive rates (failed attempts marked as successes) near zero, against 45 percent or more for WebVoyager's judge and 22 percent or more for WebJudge's, and agreement with human labelers as high as humans' agreement with each other, tested on CUAVerifierBench. These are the authors' own results, and only the talk description and the paper's abstract were reviewed.

Key takeaways · AI-distilled
  • Browserbase's Miguel González Fernández and Microsoft Research's Corby Rosset report their Universal Verifier cut false positives from about half to near zero, and agreed with human labels as often as humans agree with each other.
  • Their rubric principles: grade only what the task asked, do not let one error cascade into later criteria, check the ranked screenshots as evidence, and separate failures the agent controls from ones it cannot.
  • They validated the verifier against human labels using Cohen's kappa, and argue that training against an overconfident judge produces a more confident liar rather than a better agent.
  • They also tested whether an autoresearch loop could rebuild the verifier in a day, and report where humans still mattered. The paper and the CUAVerifierBench are published alongside the Fara code.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…