← All IntelClip / AI AgentsEmerging rubric patterns: hybrid verifiers and partial credit
From Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software · ≈17:17
Two concrete techniques: deterministic verifiers generating artifacts for judges to review, and dynamic evaluation-time rubrics that bake in a model's earlier assumptions to assign partial credit.
What’s in it
- Two concrete techniques: deterministic verifiers generating artifacts for judges to review, and dynamic evaluation-time rubrics that bake in a model's earlier assumptions to assign partial credit.
Clip transcript
Um you know these are some emerging rubric judge patterns we've seen. I'll quickly skim over this. Um you know oftent times you deterministic verifiers aren't completely dead. oftentimes we use them in tandem with judges. Maybe generating an artifact for the judge to actually look over where you're maybe collecting metrics or interesting thing that we could also use is dynamic evaluation time rubrics where we're actually generating um you know we're maybe giving partial credit where we've baked in some assumptions that the models made and assume they're correct. It's like grading a test assuming like if you got the first part wrong, let's just assume it's correct, did they get the rest of the part right? That can be really important as well as well for kind of assigning credit there.
Comments
Sign in to comment.
Loading comments…