Real AI Agents and Real Work
- Source
- oneusefulthing.org
- Author
- Ethan Mollick
- Date
Context
Research checking is a concrete use for agents that can read files and execute code. Ethan Mollick gave Anthropic's Claude Sonnet 4.5 an economics paper and its accompanying data archive. He reports that it converted statistical code and reproduced findings, which he then spot-checked. His account describes work on existing data, rather than a new experiment collecting fresh observations.
The potential value is making a laborious check easier to undertake. That could allow researchers to examine results they otherwise lack time to revisit. Mollick's trials support exploring that use, while his spot checks leave open how reliably the approach would perform across a larger collection of papers.
- OpenAI's new had professionals averaging 14 years of experience design tasks that take a human expert four to seven hours, then had a separate expert panel blind-grade the AI and human answers. Humans won, but barely.
- The AI lost that benchmark mostly on presentation, not truth: poor formatting and not following instructions exactly, rather than . Those happen to be the parts of the gap closing fastest.
- Jobs are bundles of tasks. An AI that handles two or three of a professor's tasks changes the mix of the job rather than removing it, which is why a task benchmark says little about job loss.
- Mollick asked Claude to turn one memo into a deck, then another, until he had 17 nobody needed. When generation is nearly free, the scarce input becomes judgment about which work is worth doing at all.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
“That is too many PowerPoints.”
“Even small increases in accuracy (and new models are much less prone to errors) leads to huge increases in the number of tasks an AI can do.”
Checking sign-in…
Loading comments…