60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood
Source
youtube.com
Author
AI Engineer
Date
Why it matters
CI passing does not show a schema change is safe for downstream consumers. Robinhood describes automated review gates that let a majority of data PRs merge without a human, relevant to anyone facing agent-generated PR volume.
Key takeaways · AI-distilled
Robinhood splits review into layers: scripts check facts against sources of truth (schema registry contracts, owner and PII tags, registered lineage, staging performance at production volume) and block on failure; an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → reviews only what those checks miss.
In the walkthrough, a widened int-to-bigint column passed every registered lineage check, but the agent found an unregistered job copying it into an int column elsewhere. A human confirmed the job was live, and the downstream table was fixed before merge.
The agent may never self-approve grants touching sensitive data, migrations that change what data means, or its own low-confidence verdicts. It must also report 'none found where searched', never 'no consumers exist'.
As of Aug 31, 2026, Robinhood reports about a 2% miss rate, mostly rule gaps, plus 75% less manual review, about 132 engineering hours saved monthly, and about 30% fewer PR-linked incidents.
Rules improve by hand: reviewers reply to agent verdicts with reasons, and on-call updates rules weekly. One miss, a PII retag that broke a reader of the old open copy, became a rule to search for readers before columns move.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
videoStop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
videoHow One Founder Manages 20 AI Coding Agents at Once - Ryan Carson, Untangle
videoYour KV Cache Can Spot Attacks on Your AI Agents - Ari Azarafrooz, Intrinsec AI
videoWill AI Agents Pretend to Be Human to Get Past a Check? - Maty Bohacek, Stanford
