AI Engineer 2024 Keynote - What We Learned from a Year of LLMs

Eugene Yan and his co-presenters gave the closing keynote at the 2024 AI Engineer World's Fair. Their published slides say to put a product in front of users and then look at the real interactions regularly, for example daily. The slides suggest finding slices of the model's inputs and outputs, describing each slice simply, and tracking which slices have the highest volume, which the model gets most wrong, and whether each one improves over time.

Under the heading "minimize data skew," the slides add that logs should be traced to prompt or codebase versions and that model versions should be pinned. With that record, a team could attribute a change in a slice's results to a specific prompt, code or model change instead of guessing. The slides state this as a practice and give no measured result from using it.

“Have you ever attended a talk with six speakers? Me neither.”

Eugene Yan

“While it needed more preparation and coordination, it resulted in a closing keynote that was higher signal-to-noise and energy.”

Eugene Yan

“Unfortunately, our dear friend and AIE World’s Fair organizer Swyx was unable to attend in person.”

Eugene Yan