Vibe Eval
- Source
- reka.ai
- Author
- Reka AI
- Date
Context
Reka's Vibe-Eval, introduced in May 2024, tests models on 269 image-and-text prompts with reference answers. It asks models to produce responses, allowing of visual conversation beyond selecting a multiple-choice option.
Its difficult subset was partly constructed around tasks that Reka Core and some other frontier models failed. That selection matters when interpreting the scores: it deliberately concentrates on known weaknesses of the models used to guide construction. Reka's release account also describes automated grading with Reka Core and reports that it correlates strongly with human judgment. The offers a controlled way to investigate difficult visual questions, with results shaped by both prompt selection and grading.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Checking sign-in…
Loading comments…