Vibeleaderboard
← All Intel
Intel / article

Which model is best for legacy code?

Source
factory.ai
Date
From the Daily Brief

Factory's Legacy-Bench, now included in Fireworks' Specialized Intelligence Index, asks frontier models to debug and migrate code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. Scores span from 60% down to 23% across the models tested, a much wider spread than modern-language coding benchmarks tend to show. The finding that general coding strength does not carry over cleanly to older languages matters for enterprises weighing AI-assisted migrations. The most useful detail is a failure case: one model silently miscalculated a payroll deduction while still passing most of the test suite. That is the risk profile of legacy migration in one example, since the code often encodes business rules that tests only partly capture. Teams using models on this kind of work should add property checks or parallel runs against the original system for calculations that touch money. Separately, Factory reported its model router now cuts production inference costs by 63%, up from 42% in June.

Read the 2026-09-25 Brief →

Context

Factory's Legacy-Bench tests whether coding models preserve the exact behavior older systems require. Its addition to Fireworks' Specialized Intelligence Index makes that available alongside other domain-specific comparisons. The announcement describes debugging, implementation and migration tasks across six legacy language families.

These tasks expose failures that successful compilation can hide. In one reported COBOL payroll task, a generated program passed 24 of 29 tests but calculated a $275 health deduction where $125 was required, then reported success. The evaluation therefore asks whether a model preserves exact calculations, not only whether its code runs.

Terms in this piece · Glossary
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Recommended reads
Comments

Checking sign-in…

Loading comments…