A Field Guide to Rapidly Improving AI Products
- Source
- hamel.dev
- Author
- Hamel Husain
- Date
Context
Useful AI starts with the mistakes users actually encounter. Hamel Husain's field guide describes a process of reading interactions, recording what went wrong, grouping recurring failures and turning those groups into specific tests. That gives a team something concrete to improve and measure.
The review interface matters because an answer may be impossible to judge without its conversation history or supporting records. Husain recommends putting that beside the output and making notes easy to capture. The resulting failure categories determine the measurements, so the dashboard reflects the product's problems.
- At Nurture Boss, reading real conversations showed the assistant botched relative dates like "two weeks from now" 66% of the time. Targeted tests for that one class took the success rate from 33% to 95%.
- The bottom-up method: write open-ended notes on bad traces, have an cluster the notes into a failure taxonomy, then count frequencies. Three failure modes accounted for over 60% of all problems.
- Generic dashboards do damage twice: teams feel data-driven while tracking numbers no user cares about, and optimizing many dimensions at once means none of them get .
- Prompts are just English, so routing a learning designer's expertise through slides and into an engineer's prompt adds a translation step for nothing. Give domain experts an admin UI that edits the live prompt.
- Build the annotation tool for your domain: all context on one screen, one-click correct/incorrect, hotkeys to move between examples. Every extra click is why systematic review quietly stops happening.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
“Teams with thoughtfully designed data viewers iterate 10x faster than those without them.”
Hamel
“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.”
Bryan Bischof
“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”
Phillip Carter
“The key metric for AI roadmaps isn’t features shipped – it’s experiments run.”
Hamel
Checking sign-in…
Loading comments…