Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog
Source
youtube.com
Author
AI Engineer
Date
Context
An AI application can return a technically successful response that fails the user's task. This monitoring talk, from a Datadog advocate at the AI Engineer World's Fair, argues that the classic service-health measures of latency, errors, traffic and saturation show an app is up but not whether it is right. Its quality metrics include relevance and completeness, which ask whether an answer addresses the request, and retrieval quality, which asks whether the system found useful supporting material. Monitoring those properties alongside operating costs helps distinguish an available service from one delivering useful answers. The description does not give thresholds or measured results.
Key takeaways · AI-distilled
Petzel argues the classic golden signals (latency, errors, traffic, saturation) are still necessary, but GenAI apps break their assumptions: a fast, error-free response can still be wrong.
The three hidden cost drivers she names are tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → creep, model drift and uncached calls, and she recommends a four-level tagging strategy so spend can be attributed.
Safety monitoring covers prompt injectionAn attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.Full definition →, PII leakage, toxicity and jailbreaks; quality monitoring tracks hallucinationWhen a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.Full definition → rate, relevance, user satisfaction, completeness and RAGRetrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.Full definition → retrieval quality.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
prompt injection — An attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.