Human-agent collaboration is changing, becoming more visual. The agents most teams ship today still wait for us to type a paragraph to explain what we're looking at. They cannot see a screen, navigate a UI that changes, or recover when an application throws an unexpected modal. That is the architectural gap between agents that demo well and agents that work alongside real teams in real software. Perception agents close it: they see and use computers the way people do, reason about what they see, and act with clicks and keystrokes. Speaker: Antje Barth — Member of Technical Staff, Amazon AGI Lab Antje is an AI product leader, keynote speaker, O'Reilly author, and co-instructor of Generative AI with Large Language Models with DeepLearning.AI. X/Twitter: https://x.com/anbarth Timestamps 0:00 Introduction to the AI Engineer World's Fair 0:43 The Evolution of AI Agent Capabilities 1:15 The Problem: Why Agents Struggle with Real Work 2:26 Understanding the Gap: Reliability and Trust 4:36 Why Coding Agents Succeeded: The Role of Verification 6:27 The Challenge of "Messy" Knowledge Work 7:29 How Humans Collaborate: The Power of Shared Context 9:22 Introducing Perception Agents: Perceive, Plan, Act 11:36 Why Perception Agents Matter: Closing the Loop 13:23 Open Source Harness: Annotation and Verification 16:48 Multimodal Perception: Beyond the Screen 19:30 Call to Action: Building Together Quotes "We taught computers to use computers... but we didn't solve the actual work." (1:15) "The real work lives within the seams of all of those different applications." (2:03) "If your agent one in four times deletes a database, you will never touch that agent again." (4:11) "You don't necessarily need a bigger brain. What you need is this shared context." (8:35) "We want to build AI that makes all of us smarter together." (20:07)