Building A Real Time Copilot For Global Insurance Company
Source
scale.com
Author
Scale AI blog
Date
Why it matters
Shows a post-trained open model beating the production model and a frontier model on a narrow enterprise task at roughly a quarter of the inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → cost. Splitting the task into separately evaluated subtasks keeps gains from hiding failures.
Key takeaways · AI-distilled
Scale post-trained Nemotron for an insurer's live French-language roadside-assistance calls, splitting the copilot into fact extraction, coverage recommendation and an over-promising guardrail, each evaluated separately.
On synthetic calls, fact extraction rose from 76.1% to 93.8% after SFT and RL, against 62.6% for the production model; coverage recommendation reached 95.6%, above the best tested frontier model's 92.2%.
Scoring was strict: fact extraction needed all three fields exact, coverage needed the full benefit set, and the guardrail had to match every check in a call, where violations occur in only about 4% of checks.
The recipe was SFT on the model's own correct trajectories, then RL with GSPO rewarding exact matches. Training cost about $890 for fact extraction and about $230 for coverage, according to Scale.
Fact extraction cost about $3.00 per 1,000 queries versus $12.10 for the production model. A lower reasoning setting still scored 91.9% at $1.80 and cut latency from 26 to 11 seconds. Real-call evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → comes next.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
