AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Shows how Lyft, Vodafone, and LATAM Airlines structure customer-facing AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → systems in production, including self-serve agent platforms, frontline copilots, and using evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → to align engineers with domain experts on agent behavior.
Key quotes
“LATAM Airlines saw this with Concierge. Initially, 13% of messages were classified as out of scope. After reviewing the conversations, the team found that 95% were legitimate passenger needs the agent had not yet been designed to handle, including check-in and baggage questions. Adding a customer-care specialist reduced the out-of-scope rate from 13% to 1%.”
LangChain
“Lyft facilitates 79 million trips each month, while AI Assist handles roughly 270,000 monthly interactions across seven or more production agents. The system has achieved a 65% deflection rate and a 35% AI resolution rate.”
LangChain
“This approach reduced the time required to develop an agent from roughly six months for Lyft’s first driver agent to about two weeks for a new configurable agent.”
LangChain
“Lyft’s first LLM-generated customers were too articulate, patient, and cooperative, producing offline pass rates above 90% that did not reflect production behavior. Real users often write in fragments, omit context, repeat themselves, or arrive with a specific goal such as securing a refund or bypassing the agent.”
LangChain
“Super TOBi is the agentic evolution of Fastweb + Vodafone’s existing chatbot. It now serves nearly 9.5 million customers across the Customer Companion App and voice channels, handling use cases such as cost control, active offers, roaming, sales, and billing.”
