How METR defines long-horizon via human time thresholds

but I think it's really important to actually define what the time horizon here actually means. Uh we've gotten this data from one of the most common benchmarks out there that you've probably heard of for time horizons. I'm sure you've seen in your Twitter feed all the time. It's um comes from meter. Uh and meter kind of has one response or answer to this really important question about how do we actually define long horizon? Um, it's really important to understand because what we consider long horizon a year ago probably isn't really long horizon in our definition today. And what's long horizon today probably won't be long horizon in a year or two. Um, and I think that gets to our first point which is that long horizon is really kind of a scalar metric. Uh, it's useful for kind of measuring relative tasks like one task might be more long than another, but it's really hard to define into kind of a binary category of this task is long and this task is not. Uh, especially as the kind of scope changes over time. So I think the first way we can talk about defining this is how meter kind of looks at it which is human horizon meaning can we use humans as a benchmark of oh this task takes humans a certain amount of time so if AI agents can do that then they've reached this certain critical uh level of kind of a time horizon uh and the way meter kind of does it is they have thresholds for the tasks they care about. So, you know, they have a 50% threshold, meaning, you know, if a certain model reaches a 16- hour threshold on this benchmark, that means it can achieve tasks with a 50% success rate that take a human 16 hours. And, you know, there's a really rigorous methodology of how they actually measure how did it take human 16 hours, but we will kind of avoid some of those details. Um, the