We’re demonstrating how frontier models have continued to improve in realistic…
- Source
- x.com
- Date
We’re demonstrating how frontier models have continued to improve in realistic mental health conversations with MentalHealthBench. This new open benchmark was built with input from more than 80 mental health clinicians. We’re releasing it openly so other researchers can examine the methods, run their own evaluations, and build on the work.
Most mental health benchmarks focus on emergency situations. MentalHealthBench is designed to cover the full spectrum of mental health conversations that people bring to AI - from everyday support to more acute crisis scenarios.
Context
Most existing benchmarks for mental-health-related AI conversations focus narrowly on emergency situations, according to OpenAI, which released MentalHealthBench, a new open built with input from more than 80 mental health clinicians. The benchmark is designed to cover the fuller range of conversations people actually bring to AI, from everyday support through more acute crisis scenarios, rather than testing only how a model handles a clear emergency.
OpenAI says it is demonstrating that frontier models have continued to improve on realistic mental health conversations using this benchmark, and is releasing it openly so other researchers can examine its methods, run their own evaluations, and build on the work rather than relying on OpenAI's own scoring.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleA new benchmark for evaluating patient-facing health AI agentswww.amazon.science
articleEvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language ModelsXinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert KirkarticlePiloting the world's first double-blind AI evaluationsdeepmind.google
Checking sign-in…
Loading comments…