Vibeleaderboard
← All Intel
Intel / video

Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

Source
youtube.com
Author
AI Engineer
Date

Context

An AI application can return a technically successful response that fails the user's task. This monitoring talk, from a Datadog advocate at the AI Engineer World's Fair, argues that the classic service-health measures of latency, errors, traffic and saturation show an app is up but not whether it is right. Its quality metrics include relevance and completeness, which ask whether an answer addresses the request, and retrieval quality, which asks whether the system found useful supporting material. Monitoring those properties alongside operating costs helps distinguish an available service from one delivering useful answers. The description does not give thresholds or measured results.

Key takeaways · AI-distilled
  • Petzel argues the classic golden signals (latency, errors, traffic, saturation) are still necessary, but GenAI apps break their assumptions: a fast, error-free response can still be wrong.
  • The three hidden cost drivers she names are creep, model drift and uncached calls, and she recommends a four-level tagging strategy so spend can be attributed.
  • Safety monitoring covers , PII leakage, toxicity and jailbreaks; quality monitoring tracks rate, relevance, user satisfaction, completeness and retrieval quality.
Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
  • RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
  • prompt injection — An attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…