Transcript
Sorry, not that kind of naked model. But the disappointment you're feeling right now? That's exactly how Kyle Kingsbury feels about LLMs.
Kyle Kingsbury is one of the best systems engineers alive. His Jepsen project spent a decade methodically proving that distributed databases didn't work as advertised. That CockroachDB, MongoDB, Redis, and dozens of others made consistency guarantees they couldn't keep. He published the results, the vendors fixed the bugs, and the entire industry got more honest. Jepsen is a masterwork of applied skepticism.
Last week he published a 32-page essay called "The Future of Everything is Lies, I Guess: Bullshit About Bullshit Machines." It's beautifully written, deeply researched, genuinely funny, and wrong about the most important thing.
His observations are correct. His conclusion is not.
A note on scope: this essay addresses Kingsbury's technical claims: that LLMs are unreliable bullshit machines incapable of producing trustworthy output. His broader concerns about labor displacement, information ecology, and cultural impact are real, separate questions that deserve their own essays. I'm not dismissing them by not addressing them here. I'm addressing the architecture question: does model unreliability make useful systems impossible, or does it make them an engineering problem? I think it's the latter. Kingsbury's essay assumes the former. That's where we disagree.
Testing the engine on a bench
Kingsbury's essay is structured as a catalogue of LLM failures. He asked Gemini to apply materials to a 3D bathroom rendering. It forgot the toilet and changed the room's shape. He asked Claude to do image-to-image transformation. It produced thousands of lines of JavaScript creating an incomprehensible garble of nonsense polygons. He asked ChatGPT to put white patches on a blue shirt. It changed the color, moved the patches, deleted them. He watched a colleague's LLM claim to download stock data and produce a graph of randomly generated numbers.
Every single one of these failures is real. I've seen failures like them. Everyone building with LLMs has. Kingsbury is not making anything up.
But here's what's happening in each example: a human sits in front of a raw language model, types a request in natural language, and watches it fail.
No skill file telling the model how to approach the task. No deterministic tool handling the parts that require precision. No resolver routing the request to the right capability. No harness managing context, enforcing safety, or constraining behavior.
He's testing the engine on a bench and concluding that cars are unsafe.
The architecture question
Some terms for readers who haven't read the first essay in this series. A skill file is a reusable markdown document that teaches the model how to approach a task — a procedure, not a prompt. A resolver is a routing table that tells the model which document to read for which task. Deterministic code is software that produces the same output every time — SQL queries, API calls, math — the parts the model should never touch. A harness is the thin conductor that runs the model in a loop, reads files, and manages context. Together: thin harness, fat skills.
Kingsbury's central claim is that LLMs are "bullshit machines." They confabulate, they're chaotic, they're vulnerable to manipulation, and they can't be trusted. He arrives at this by testing models in isolation, the way you'd test a function: input in, output out, evaluate the output.
The people having Kingsbury's problems are the ones doing exactly this: chatting with a raw model and expecting reliable output. The people who aren't having those problems built harnesses. Not because harnesses make the model trustworthy. Because harnesses make the system trustworthy, even when the model inside it is not.
Kingsbury knows the harness exists, incidentally. He cites the Claude Code source leak: 512,000 lines of engineering that Anthropic built around their own model. Even the makers of the best LLM in the world don't trust the model naked. They wrapped it in live repo context, prompt caching, purpose-built tools, session memory, and parallel sub-agents. That's 512,000 lines of evidence that model reliability is an engineering problem, not a philosophical one.
But I want to be honest about something: 512,000 lines is not simple. The pattern I'll describe: skills, resolvers, deterministic code, testing— is conceptually clean. Robust implementation is real engineering. Harnesses fail. Skills encode wrong procedures. Verification layers are only as good as the invariants they check. The claim is not that harnesses make everything easy. The claim is that they turn model unreliability from a reason-to-stop into a problem-to-solve. That distinction matters.
The stock data example is the clearest illustration. An LLM claimed to download stock prices and produced a graph. The data was random. Kingsbury presents this as evidence that LLMs lie. But what actually happened? So