agents we're building were doing scientific work. Uh the existing benchmarks we saw at the time did not measure the tasks relevant to this category of work. Um they mostly measured things in a Q&A settings, like what what do in a kind of academic way, or they weren't sufficiently focused on the experiment type. This is an actual screenshot of Anthropic's model card at the time that we built this benchmark. So, we built one. It's called spatial bench. Um last December, there's 146 problems. They spanned the different kits I talked about or attempted to. And then they spanned all those different tasks that I talked about as well. So, the thing that we found at this time, and still to an extent is true today, is the the grading of these end outcomes in biology uh is too sparse because the models are pretty bad. So, you have to break things up into manageable chunks to get some semblance of verifiability. And that's kind of induced by um sticking to these little components of like that DAG, that analysis DAG. Um getting data to a state where it would exist right before a scientist or theoretician could do work on it, and then um figuring out what the ground truth would be in that context. So, a single evaluation kind of looks like one or more data nodes, again like a matrix of numbers, high content image, something like this. A task prompt carefully describing some scientific goal, configuration for a grader, and then a deterministic grader, so a Python function. If you guys notice, this looks a lot like SweetBench. We borrowed a lot of the early ideas and tried to extend them as much as possible. Evaluation ends up looking like this. It's a lot of JSON. And we ended up identifying properties of like what we thought good biological tests were. Um little little different from code, and we built on these over time, but they still hold up. They got to be verifiable. You have to be able to check the success condition with a function. Um nothing's changed there. We'll get into some rubric stuff later, but still holds. Durability is particularly important. Science does not admit clear ground truth. Um if you are lazy with your ground truth construction of the task, a possible valid analysis path can come with the correct answer, um and you'll fail it uh incorrectly. So, you got to make sure you're reasoning about something that's somehow invariant across analysis paths.