Vals AI benchmarks show models give incomplete advice on high-stakes questions
- Source
- video.twimg.com
- Author
- Vals AI
- Date
Context
Vals AI, a company that runs independent tests of AI models, has built three benchmarks that ask whether chatbots can help ordinary people with high-stakes problems: Immigration Bench, Access to Justice and Financial Coaching Bench. On its podcast The Bench, two Vals researchers say the main failure across all three is not wrong advice but incomplete advice, and that the people with the most to lose get the least complete answers (Vals AI on X).
Each starts from realistic questions written or checked by practitioners such as immigration experts, legal aid experts and financial counselors, who also write a checklist of what a complete answer must include. The model being tested talks with a simulated user who holds hidden facts and reveals them only when asked, so the score also reflects whether the model asks the right questions. An AI grader then marks which checklist items the conversation covered, and the score is the share met. The researchers chose this format because real questions arrive vague and underspecified, and people often do not know which details matter.
The published results support the incompleteness finding in all three benchmarks, but the gap by who is asking is clearest in immigration. In the immigration report, built with the Talent Mobility Fund, the best model, Claude Fable 5.1, covered 75.4% of checklist items and fully answered only 26.3% of questions; across all models, international students' questions averaged 85.6% while those from people on DACA or Temporary Protected Status averaged 51.1%. On Access to Justice, which Vals launched with NextLadder Ventures on November 10, no model reached the 80% that Vals set as the bar for a reliable first point of contact, and models met 78% to 87% of criteria for explaining the law but only 30% to 50% for pointing someone to specific local help such as a legal aid office. On Financial Coaching Bench, about 95% of failed items were advice the model left out or only partly gave, and Vals says the default models behind free ChatGPT and Google's free AI products ranked 25th to 28th of 28. There the gap ran by product rather than by client: coaches met 73.5% of checklist items for clients earning under $30,000 and 69.1% for those earning more, and Vals found no model that scored reliably worse for the lower-income group.
That pattern is why incompleteness matters more than it sounds. A chatbot that confidently explains the general rule but omits a filing deadline or a local legal aid office gives the reader no sign that anything is missing. One researcher's example was a deadline for international students on OPT work authorization that was cut from 60 days to 30 while the benchmark was being built; he says most models still answered 60. The researchers also argue that better general models do not fix this automatically, citing newer OpenAI models that scored below older ones, a regression the financial leaderboard shows. In the immigration test, the best answer to each checklist item across all 25 models would have scored about 92%, so much of the missing knowledge exists somewhere but no single model applies it reliably. These are Vals's own benchmarks discussed by their authors, the scores depend on an AI grader and change as leaderboards update, and this explanation draws on only the first 30 minutes of the episode.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Checking sign-in…
Loading comments…





