Building An Agent For Our Own Docs

In a post first published on January 21, 2025 and updated in September 2026, an engineer at ElevenLabs describes the voice agent the company embedded in its own developer documentation. He says it handles over 80% of user inquiries across about 200 calls a day (elevenlabs.io). That figure needs careful reading. Here, a call counts as a success if the agent either answered with relevant information or sent the caller to the right documentation page or support channel. The judge is another LLM that reads each finished transcript against criteria ElevenLabs wrote, and calls that never got past one turn are left out. The post itself cautions that an evaluation LLM will not be correct 100% of the time.

To check that judge, the team had a person grade 150 conversations against the same criteria. The human and the LLM agreed on 81% of cases about whether the inquiry was solved. They agreed on 83% of cases about whether the agent's answers stayed within the knowledge base or went beyond it, which is how the team checks for made-up product claims. The human review found that 89% of relevant support questions were answered or redirected correctly. All of these are ElevenLabs measuring its own product, with a small human spot check. The post reports no outside evaluation.

The build is simple. A system prompt tells the agent to answer only from the documentation it is given. The knowledge base is a summarised but still large version of all ElevenLabs documentation, about 80,000 characters, plus FAQs and clarifications. Three tools let the agent send the caller to a docs page, open an email to support, or follow an external link to sales or the company's Discord. Much of the prompt deals with speech rather than content: write email addresses the way they are pronounced, avoid bullet points and long lists, never read out code, and answer in a couple of sentences.

The agent did well on clear, specific questions, such as whether the API has an endpoint for deleting a voice. It was less helpful with account problems, pricing and discount requests, and vague issues. Even when prompted to ask clarifying questions, the post says, the LLM usually prefers to answer with something from the documentation that might seem relevant. For debugging, the author concludes that voice is not the right medium for sharing code, so the agent should send callers to examples or to human support. Some callers just wanted to try out different languages or talk about unrelated topics, and prompt guardrails held most of the time, but not always.