An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks
Source
Chandimal Adikari, Nandika Herath
Author
Chandimal Adikari, Nandika Herath
Date
Key takeaways · AI-distilled
The study tested three specific cost-efficient models — Gemini Flash 3, GPT-5.4 mini, and Claude Haiku 4.5 — generating 7,593 Java Spring Boot service methods across 992 problems and 8 model/tool/prompt configurations, with iteration and hardcoded answers explicitly forbidden.
Correctness and reliability point opposite ways: methods classified as genuinely computing an answer were submitted least often of the eight outcome classes, yet were correct 19.3% of the time — the highest rate of any class.
Eight of the 992 problem statements were deliberately given missing/incomplete specification inputs, specifically to see how each model handles unclear requirements rather than just well-posed problems.
The authors flag their results as provisional: a single run per configuration, partial agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → coverage, single-pass timing, purely syntactic classification of outputs, and a real risk some problems existed in training data.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Why it matters
Warns against trusting cheap models for enterprise codegen based on 'it compiles and looks right' checks alone; correctness and structure diverge sharply here.
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleCode Transformation Rule Synthesis using LLMs: Potential and LimitsAxel Allain, Aymeric Blot, Djamel Eddine Khelladi, Mathieu Acher
articleA Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMsHaruka Tokumasu, Masanari Kondo, Alexander Serebrenik, Dong Wang, Kei Koyanagi, Kotaro Noguchi, Naoyasu Ubayashi, Yasutaka Kamei