Intel / article

Common pitfalls when building generative AI applications

Source: huyenchip.com
Author: Chip Huyen
Date: Published Jan 16, 2025

Context

An AI evaluator can give an application improving scores while human users see worsening results. Chip Huyen's account of common development pitfalls recommends checking automated judgments against regular human review. The evaluator is itself an AI application whose behavior depends on its model, instructions and use case.

Human reviewers therefore do more than assign another score. Looking at actual outputs can expose a mismatch in the evaluator or a change in what users are asking. Clear review guidelines also make those judgments more consistent and can help improve the application's own instructions.

Key takeaways · AI-distilled

Users often do not want the thing you are optimizing. A meeting-summary team argued over three- versus five-sentence summaries; users did not care about the summary at all, only the action items assigned to them.
LinkedIn's -fit bot taught them users want helpful, not correct. "You are a terrible fit" can be accurate and still useless. People want the gaps named and a path to closing them.
Intuit's tax bot flopped because users hated typing into a blank box. Adding a few clickable suggested questions each turn cut the friction and turned feedback positive. A UI fix, not a model fix.
Early progress badly misleads. LinkedIn hit 80% of the experience it wanted in one month and needed four more to clear 95%; one startup found 0 to 80% took as long as 80 to 90%.
Keep daily human review of 30 to 1,000 outputs next to your judge. If human scores fall while the judge's rise, the judge is broken. Judges are AI applications too, and they drift.

Terms in this piece · Glossary

agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

““We solve the problem” and “We use generative AI” are two very different headlines, and unfortunately, so many people would rather have the latter.”

“It’s easy to build a demo, but hard to build a product.”

“Manual inspection of data has probably the highest value-to-prestige ratio of any activity in machine learning.”
Greg Brockman

“the journey from 0 to 60 is easy, whereas progressing from 60 to 100 becomes exceedingly challenging”

Read the source huyenchip.com