Production traces showed 95% of out-of-scope messages were real needs

LATAM Airlines' Concierge agent, which helps passengers in the LATAM app find flights, explore destinations and book hotels or rental cars, was classifying 13% of messages as out of scope. According to LangChain's case study, the team first assumed users were testing the agent or asking unrelated questions. When it reviewed the recorded production conversations, known as traces, it found that 95% of those messages were legitimate passenger needs the agent had not been built to handle, including check-in, baggage, LATAM Pass, special services and travel requirements.

Concierge uses a supervisor that hands work to specialist agents for flights, bookings, destinations and other tasks. LATAM added a customer-care specialist for the rejected requests, and the out-of-scope rate fell from 13% to 1%. LangChain reports that the return rate improved by 6%, and the article adds that about 12% of daily messages now go through the new specialist. These figures come from LangChain's write-up about a customer that builds on its tools, and the write-up does not define how the return rate is measured.