What's the Best Programming Language for Coding Agents?
- Source
- x.com
- Author
- Dan Luu
- Date
Context
Dan Luu's essay tests the claim that concise, dynamically typed languages are cheaper for AI coding agents, and finds it does not hold up across his tasks. He gave agents two jobs: implementing a Zstd decoder (a program that unpacks compressed data) from the specification, and a Pandoc-based document-conversion task. The Zstd agents were not given the tests; the Pandoc agents received development tests and were scored on a separate holdout set.
He reports that the apparent advantage of dynamic languages appeared at medium reasoning effort on Zstd but became mixed at the highest effort, and that results differed again on Pandoc. In the Zstd runs, 36 of 40 medium-effort and 5 of 40 ultra-effort Clojure programs failed tests because byte conversion throws on values from 128 to 255. He calls this a real result for the best publicly available GPT model, since failures like this cost time and even when tests catch them. He also warns that his own are likely flawed and that many more tasks would be needed to rank languages, so two tasks challenge a universal efficiency rule without establishing a best language.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Checking sign-in…
Loading comments…