Intel / article

The Shape of the Thing

Source: oneusefulthing.org
Author: Ethan Mollick
Date: Published Mar 12, 2026

Context

Automating code production changes where people direct and evaluate software work. Ethan Mollick's account of a software-factory experiment describes humans writing product roadmaps and reviewing finished products while agents handle implementation and testing. Human responsibility remains at those boundaries.

StrongDM, a security software company, built the experiment around testing agents that try the software in a simulated customer environment, building it as needed, and loop with the coding agents until the results satisfy the AI. Its rules bar humans from writing or reviewing the code, so the roadmap and the final product check are where people still apply judgment. It is one small team's experiment, not evidence that the approach works generally.

Key takeaways · AI-distilled

The software factory's economics are the real disclosure: each engineer is expected to spend roughly their own salary on , at least $1,000 a day. Human labor becomes the cheap input in the loop.
Testing agents don't just run tests — they build the customer environment, including a simulated Slack where fake customers file requests, then feed results back to the coding agents until the agents are satisfied.
The gap is now inverted: on graduate-level science questions, PhD students score about 34% outside their field and 70% inside it, while the best models score 94%.
On GDPval, where industry experts compare AI work to experienced human work on real tasks, current models match or beat top human performance 82% of the time.
One February week showed the pattern: a fictional AI-disruption scenario moved real stock prices, a 40% layoff was blamed on AI it probably had little to do with, and a government-lab fight erupted over usage rules.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

Key quotes

“This is an era of managing AIs, rather than working with them.”
Ethan Mollick

Read the source www.oneusefulthing.org