Vals AI

Vals AI released MysteryMechanism, a benchmark testing whether agents can rediscover sealed mathematical mechanisms through bounded experiments; GPT-6 Astra scores roughly 20 percentage points above GPT-5.6 Sol.

Vals AI, with marimo and CoreWeave, launched the RSI Index, a third-party benchmark scoring frontier models on AI-research tasks against the best published human results; current models can attempt the work but remain far from the human frontier.

Vals documents giving agents two fixed Lean statements for the N=7 Thomson problem, where the pentagonal bipyramid should minimize Coulomb energy on a sphere, and what the resulting formal proof contributes. The excerpt does not show the full results.

Vals tests Chinese, American and other models on hundreds of China-related political questions across languages, contexts and harness setups. Some Chinese models openly criticize the government, so the lean is less uniform than assumed.

Vals AI's new CUA-Bench tests whether models can control a keyboard and mouse across six real-time games, arguing current agents still lag on continuous, video-driven action compared to text-based tasks.

Vals recounts how it built Vibe Code Bench, a benchmark for building complete applications from scratch, including why coding is the key domain to measure and the design decisions and lessons behind it.

OpenAI launches Astra for Law, a legal research agent validated against Vals AI's Legal Research Bench, where it beat GPT-6 Astra with web search at every price point (90.0% vs 84.1% weighted rubric, 53.7% vs 39.6% all-pass).

Vals AI launched the Vals Web Search Index, a third-party benchmark run with the same settings for every provider, with Exa, Keenable, Parallel and Tavily as partners. It aims to fix contamination, realism and answer-leakage problems.

Ant Group's finance-tuned Ling 3.0 Flash Fin scores near-frontier on Vals AI's Finance Agent v2 at $0.045/task, 25-75x cheaper than top models, though it trades away coding and general capability.

Vals AI runs SWE-Bench Multimodal on 15 models in mini-swe-agent without internet. Opus 5.5 leads at 70.2%, cost per task does not track score, and enabling web access let agents copy upstream patches, inflating results by 21 to 35 points.

Vals AI launched Terminal-Bench 4.0, 66 new terminal-based tasks (shipping a service, proving a theorem, training a GPU kernel) each estimated at 4 hours of expert work, graded strictly pass/fail via full verifier suites and averaged across three runs.

Vals AI traced a jump in xAI's benchmark ranking to an SDK fix: the native xAI SDK wasn't returning encrypted_reasoning_content, which carries reasoning forward between turns, silently losing that context.

Vals AI describes agents built on Claude Opus 5.5 that designed one new compound and identified a 1999 material predicted to be room-temperature antiferromagnetic semiconductors. It shares calculations, code and known caveats.

Vals AI released Vals Index v2.1, swapping Terminal Bench 2.1 for Terminal Bench 4 and adding its proprietary Tax Agent Benchmark as a separate GDP-weighted category.

Vals AI's testing of nine models across 648 simulated 10-turn teen conversations found critical safety failures in 27.5% of chats, most surfacing only after the first reply, undercutting single-turn safety evaluation.

Vals introduces its Web Search Index, arguing BrowseComp and HLE reward more than search and miss production tasks. It tests agents with specific search tools on real knowledge-worker problems, addressing contamination and answer leakage.

Vals AI reports Keenable's search API with GPT-5.6 Sol topping its Web Search Index at 49.1%, leading finance. The thread breaks down cost per task, search share of spend, source ranking rates, and latency against native OpenAI search.

Vals argues model routers are a new primitive, citing the Vals Index where five models hold top spots across eight benchmarks and none ranks top three on all. Choosing a single model means giving up performance on some tasks.

Vals benchmarks AI-text detectors and general LLMs on private human documents and model rewrites. General models lead on balanced accuracy, and detectors differ sharply in how many rewrites they catch versus how many human texts they wrongly flag.

Vals AI's index now ranks Claude Opus 5.5 first, with Anthropic holding the top three spots; Opus 5.5 also became the first model to beat the reference score on the long-horizon agentic LM Training protocol.

Vals AI released Vals Index v2.2, updating four constituent benchmarks (Finance Agent, Excel Modeling, Code Migration, Harvey's LAB) with new judges, stricter grading and a Pi agent harness. Sonnet 5.5, Opus 5.5 and Gemini 4 Argon now lead.

Vals tests chatbots across 648 multi-turn teen conversations. 27.5% included a serious safety failure, often late in the exchange after earlier correct behavior, and teen-specific instructions cut failures from 31.1% to 11.5%.

Vals AI's RSI Index tests 25 models in off-the-shelf harnesses on four autonomous ML research tasks. Opus 5.5 beats a published LM-training reference. Muse Wonder Full ranks #9. Model profiles show differing research behaviors and failure modes.

Vals AI benchmarks Inception's Mercury Decide against 13 systems. It ties frontier accuracy on SEC claim verification at the lowest cost measured, beats Jev on LegalBench, and automates 94% of claims at 0.8% error, though Jev stays faster at scale.