What We've Learned From A Year of Building with LLMs

Real product failures can become repeatable checks for future changes. In their 2024 engineering guide, Eugene Yan and coauthors recommend turning actual inputs and outputs into evaluations with explicit expectations.

Those examples can be rerun when a team changes a prompt, retrieved context or another part of the system. Simple requirements can use direct assertions; generated code can be checked by execution. Human review remains useful for failures that are hard to express mechanically. The resulting test collection connects development decisions to the work customers actually ask the product to perform.

“Vector embeddings do not magically solve search.”

Aravind Srinivas

“Nearest Neighbor Search with naive embeddings yields very noisy results and you’re likely better off starting with a keyword-based approach.”

Beyang Liu

“When a measure becomes a target, it ceases to be a good measure.”

“the 10m context kills RAG.”

Yao Fu