Let's Let's start to improve and learn more about this benchmarking problem." So, we jumped to human verification long horizon extension. I'm going to quickly breeze through these. So, human verification is incredibly important in science. Science is not in the clear ground truth. Uh after watching trajectory data from multiple rounds of model releases, circa like January to March of this year, um we really realized a lot of our assumptions were pretty bad. Um and in the absence of like a canonical answer, uh having a bunch of scientists grade each other's work ended up being like the best proxy. So, I'm going to look at one one issue to highlight exactly what I'm talking about. Um this problem ambiguity. A task might ask an agent to split a gene list into two groups of activity, microglial activation, oligodendrocyte inflammation, just like biological categories of things. Score the cells, find neighboring oligodendrocytes around some region using an appropriate radius, compute a Spearman correlation at two time points. Uh as you can probably clearly deduce, the original problem statement creates a host of open choices. How do you split the gene list? How do you count what inflammatory genes are? It's like somewhat ambiguous word. How do you normalize the data? Um what what what the hell is an appropriate radius? How do you pull the counts within the selected radius? Um These are all problems that pointed to tasks that were bad, that only became revealed with human verification. Another issue is just like a lot of people in bioinformatics canonically have used like numerical thresholds to QC stuff. Just like completely arbitrary stuff. Um cool thing about evaluation like coding is it forces you to reason about things more rigorously than you would when you're doing the thing yourself. If you have to teach a machine to do it, uh you you might be picking out some structure that's more important or more durable than what you were doing if if you're just doing it on your own. So, we just found a lot of these numerical thresholds to be like that. Uh I'm not going to get into this. Um after two rounds of human attempts, we produce a verified subset of the benchmark. Uh we we publish it. That was fun. Uh and then we also tried to increase the time horizon. So, I want to be clear the frontier of knowledge is still not quite there with biology. Like the labs are starting to catch up with the post training, but we kind of want to stay ahead. Um so, we built a benchmark that we thought would recapitulate like really difficult true work. Um so, we built a space much longer. Um real biological tasks are messy. They use lots of different experiment types. They are use the whole workflow, don't use little chunks. They are uh tasks every step is interpreted against the experimental design or contextualized with some prior literature and the original goal of what you're doing in the first place. Um so, we built a bank of these tasks that are really trying to simulate the result sections of entire papers or the kinds of decisions you make in practice in industry to make a go no go decision on a drug program. Um these got these tasks took like a week for a group of three people to make each. Taught us a bunch of stuff. Um example is can an agent reconstruct a like metastatic niche in a tumor? Um if you have like a tumor biopsy and a bunch of metastatic biopsies from like where it metastasized and spread across the body, can it use like both the genetics and mRNA of the metastatic lesions and the tumor to like find the part of the tumor that initially seeded the metastatic growth and let it spread? From that you can figure out like, "Hey, what parts of the tumor are more like genetically fit? Which ones actually cause problems?" And construct targeted medicines to nip them in the bud. Like for example, this is one of the benchmark uh evals in the long horizon set. None of the models get this right. Um but they're getting there. As As imagine with these long horizon extensions, uh verifiable reward at the end are like somewhat uninformative. So, we we're starting to play with rubrics, uh constructing these choke points. If you can imagine like the set of analysis paths is inducing some sort of tree. Um there are nodes that are invariant with respect to yeah, different paths, and you can use these to build rubrics um using knowledge of how the task work. Uh we we we're playing with these. We noticed that um they're associated with the verifiable outcomes, uh which is exciting, but they're loosely correlated numerically, um making us not fully have confidence in them for things like RL or benchmarking. Um a lot of lot more work to do here still.