It gives practicing engineers a concrete mental model for the transition from working alongside AI to orchestrating fleets of agents, groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → in a real 'software factory' case study where humans no longer write or review code.
Key takeaways · AI-distilled
The software factory's economics are the real disclosure: each engineer is expected to spend roughly their own salary on tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →, at least $1,000 a day. Human labor becomes the cheap input in the loop.
Testing agents don't just run tests — they build the customer environment, including a simulated Slack where fake customers file requests, then feed results back to the coding agents until the agents are satisfied.
The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → gap is now inverted: on graduate-level science questions, PhD students score about 34% outside their field and 70% inside it, while the best models score 94%.
On GDPval, where industry experts compare AI work to experienced human work on real tasks, current models match or beat top human performance 82% of the time.
One February week showed the pattern: a fictional AI-disruption scenario moved real stock prices, a 40% layoff was blamed on AI it probably had little to do with, and a government-lab fight erupted over usage rules.
Terms in this piece · Glossary
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Key quotes
“This is an era of managing AIs, rather than working with them.”
