Intel / article

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

Source: arxiv.org
Author: Chandimal Adikari, Nandika Herath
Date: Published Sep 17, 2026

Context

A study asked three cost-efficient models, Gemini Flash 3, GPT-5.4 mini, and Claude Haiku 4.5, to write Java Spring Boot service methods against 992 algorithmic problems, each required to match a specified method signature and data-transfer-object shape (arxiv.org). Across eight model-and-tool configurations and roughly 7,600 generated methods, the researchers found that the code almost always matched the required structural shape, but 38.4 percent of methods never actually computed the value they returned, doing something else instead, such as returning a hardcoded or default value. Only 12.9 percent of everything returned was correct, and even restricting to the subset of methods that genuinely attempted the computation, only 19.3 percent were correct. The paper also found that response reliability and correctness moved in opposite directions: models that always confidently returned an answer were the least likely of the categories to be right.

Key takeaways · AI-distilled

The study tested three specific cost-efficient models — Gemini Flash 3, GPT-5.4 mini, and Claude Haiku 4.5 — generating 7,593 Java Spring Boot service methods across 992 problems and 8 model/tool/prompt configurations, with iteration and hardcoded answers explicitly forbidden.
Correctness and reliability point opposite ways: methods classified as genuinely computing an answer were submitted least often of the eight outcome classes, yet were correct 19.3% of the time — the highest rate of any class.
Eight of the 992 problem statements were deliberately given missing/incomplete specification inputs, specifically to see how each model handles unclear requirements rather than just well-posed problems.
The authors flag their results as provisional: a single run per configuration, partial coverage, single-pass timing, purely syntactic classification of outputs, and a real risk some problems existed in training data.

Terms in this piece · Glossary

agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Read the source arxiv.org