Making Failure Diagnosis Reliable with Pensive AI

It shows how a large data platform extends a rule-based failure classifier with AI to cover the roughly half of failures rules missed, without giving up deterministic remediation.

• Netflix's rule-based Pensive made about half of workload failures self-service, but over three months roughly half of ~7 million failure classifications landed in an unknown bucket, because each regex or Java rule matches only a log shape someone already wrote.
• Pensive AI gives the model a line-numbered log and asks only for the first and last line of the error span, then rebuilds the text from its own copy. Output stays two integers at any error length, cutting cost and latency and keeping displayed text verbatim.
• Error extraction runs on every failed step inside Maestro's 90-second hard timeout: per-log model calls run in parallel under one shared deadline, and any call that misses it is dropped so the response still returns on time.
• Full debugging runs only on demand and is cached per failed step attempt, not per user. A conditional write lets one caller generate while others wait, so the UI and MCP paths share one answer, and each streamed event carries the full text so far.
• Netflix uses its frozen legacy rules as eval labels, archives logs to S3 so inputs do not expire, and gates each pull request on inclusion and precision thresholds. Steps where no one used Pensive AI saw 321% more manual log exploration (observational, not causal).

• MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

• blogThe Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale
• blogMAPS: Netflix’s Multimodal Asset Personalization at Scale
• blogIn-House LLM Serving at Netflix
• blogGenRec: Towards LLM-Native Recommendation at Netflix

• video1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York LifeAI Engineer
• videoAI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, NetflixAI Engineer
• videoProduction Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence LabsAI Engineer

Checking sign-in…

Loading comments…