OASIS: A Rubric-Based Multimodal Assessment Platform Using Large Language Models
Source
Ameer H. Shakur, Shinyoung Kang, David Hein, Michael J. Holcomb, Huong-Tra Ngo, Aarash Zakeri, Minhan Park, Licheng Yi, Dhanush Jain, Hunter Schuler, Andrew R. Jamieson
Author
Ameer H. Shakur, Shinyoung Kang, David Hein, Michael J. Holcomb, Huong-Tra Ngo, Aarash Zakeri, Minhan Park, Licheng Yi, Dhanush Jain, Hunter Schuler, Andrew R. Jamieson
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Scoring one artifact with an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → is easy, but running rubric-based assessment reliably at scale requires infrastructure most teams build from scratch; OASIS packages that as a reusable CLI plus stack for anyone building evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → pipelines.
Key quotes
“OASIS pairs a standalone command-line interface with a canonical integrated Elephant + MAPLES stack for encounter management and multimodal grading orchestration.”
“Distinctive features include rubric-as-program compilation, progressive execution plans, content-addressable grading identity, transcript-augmented multimodal grading, explicit review state, and a shared command surface for humans and autonomous agents.”
“In production at UT Southwestern Medical Center since Fall 2023, the platform has processed more than 7,000 encounters.”
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi
articleolmo-eval: An evaluation workbench for the model development loopallenai.org
articleCompliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System MessagesJuan Yeo, Geewook Kim