How to Kill Your SaaS: Results From a 72-Entry Clone Hackathon
- Source
- x.com
- Date
🆕 How to Kill Your SaaS https://t.co/A0F60h1DcQ A lot of people talk about the death of SaaS but mostly in theoretical terms. We took a real $43,000/year SaaS we use and launched a global hackathon to clone it in a weekend, with both AI and REAL HUMAN evals, using any models any harness. 72 submissions, a lot of manual evals and ~100B tokens later, here are the results In this special @latentspacepod edition we share the data collected, what's still hard about killing nontrivial, midtier SaaS, and reveal the winners of the $10,000 Kill My SaaS contest with qualitative interviews!
Context
This contest tested a software clone against the people expected to use it. The organizers invited builders to reproduce their conference speaker-management tool, then had their events team evaluate submissions. In the accompanying account, they report that their ranked the eventual winner nineteenth. That gap makes part of the story: an automated ranking can differ substantially from the contest's final result. Organizers' account.
The $43,000 annual quote motivated the contest. It does not establish the cost of operating a replacement or a realized saving. The supported result is a comparison of submissions and judgments, with maintenance costs still outside what this account demonstrates. Contest description.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…