ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
Source
huggingface.co
Date
Context
A migrated application can compile successfully and still fail to behave correctly. IBM'sScarfBench tests that gap through Java application migrations among Spring, Jakarta EE and Quarkus. It checks whether the resulting application builds, deploys and passes behavioral tests. In the authors' reported evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, even the strongest agents achieved less than 10% behavioral success, with build success consistently higher. Configuration, dependencies and runtime behavior therefore remain part of the migration task even when the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → produces plausible replacement code.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
