On Working with Wizards

An AI finding a real mistake can still leave its other claims difficult to verify. Ethan Mollick asked OpenAI's GPT-5 Pro to critique a published paper he knew intimately. He checked and confirmed a small inconsistency involving two tables, but could not establish that every experiment the model said it had performed happened as described. The account distinguishes a verified correction from an opaque analysis.

That is the central problem behind his wizard metaphor: receiving sophisticated work with little intervention shifts effort toward judging the result. Familiarity with the original paper let him check the correction; the apparent quality of the remaining output did not supply the same assurance.

“We're moving from partners to audience, from collaboration to conjuring.”

“It is a transition from working with a co-intelligence to working with a wizard.”

“The paradox of working with AI wizards is that competence and opacity rise together.”

“Welcome to the age of wizards.”