Eugene Yan
22 Intel
Eugene Yan is a machine learning practitioner based in Seattle, currently a Member of Technical Staff at Anthropic working on safe, reliable AI systems. He previously spent five years as a Principal Applied Scientist at Amazon building retrieval and recommendation systems, and earlier led machine learning teams at Alibaba/Lazada and a Southeast Asian healthtech startup. He holds a master's degree in computer science from Georgia Tech and writes a widely read newsletter on LLMs, recommendation systems, and applied machine learning at eugeneyan.com.
Is this you? Sign in with X to claim this profile.
Intel
Special double-feature closing keynote from the 6 authors of the hit O'Reilly article on Applied LLMs.
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
An argument that chat became the default LLM interface by inertia rather than fit, and a case for interaction patterns that expose model capability more directly than a conversational text box.
9 patterns including HITL, hard mining, reframing, cascade, data flywheel, business rules layer, and more.
Structured input/output, prefilling, n-shots prompting, chain-of-thought, reducing hallucinations, etc.
Overcoming the bottleneck of human annotations in instruction-tuning, preference-tuning, and pretraining.
A synthesis of how large language models are rewriting recommendation and search systems: architectural shifts in retrieval and ranking, using models to generate training data, changing training paradigms, and the move toward unified frameworks.
Build a threat model, discover vulnerabilities, verify, triage, and patch.
How unit testing machine learning code differs from typical software practices
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
From the tactical nuts & bolts to the operational day-to-day to the long-term business strategy.
A working model for compounding output with AI: treat context as infrastructure you maintain, encode taste as configuration, use verification to buy autonomy, scale through delegation, and close the loop so each cycle improves the next.
Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.
Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
An LLM that can converse in English & item IDs, and make recommendations w/o retrieval or tools.
A sandboxed target, inputs that influence task difficulty, tools, and a grader.
Applying the scientific method, building via eval-driven development, and monitoring AI output.
How to use open-source, permissive-use data and collect less labeled samples for our tasks.
Challenges and lessons from deploying LLM experiences: evals, scalability, guardrails.