A few hundred expert-built tasks per domain raised Pass@1 on APEX-SWE from 26.6% to 34.7%. That shows small targeted datasets can meaningfully specialize an open model for domain agents.
Key takeaways · AI-distilled
On the legal slice of APEX-Agents, Mercor reports the post-trained model's mean rubric score rose from 45.1% to 51.8% and its full-credit rate from 19.8% to 25.0%.
On APEX-Accounting, mean rubric score rose from 34.6% to 41.9%, and the share of tasks where the model scored zero fell from 19% to 13%.
The legal model was trained with supervised fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → on model-generated trajectories selected by a grader, then tested on benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → tasks with no case files shared between training and evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →.
Legal gains were uneven across task types: motion drafting improved most (+23.2 pp), followed by litigation strategy (+10.3 pp) and legal research, the most common type (+7.0 pp).
The post-trained legal model read the case file more thoroughly, answering without any tool in 2.6% of attempts versus 10.6%, doubling document reads per attempt to 11.3, and checking email, chat or calendar far more often.
Terms in this piece · Glossary
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
