Clip / AI Agents

Swiss cheese model: deliberately redundant QA gates

From Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber · ≈17:39

Justifies a second, more holistic publish-ready QA gate on top of the in-loop QA, accepting redundancy to lower the odds of any single failure reaching production.

What’s in it

Justifies a second, more holistic publish-ready QA gate on top of the in-loop QA, accepting redundancy to lower the odds of any single failure reaching production.

Watch this moment

Clip transcript

So, the last step after all of that is a post-processing and what we refer to as the publish-ready QA. This is the final gate before we decide we want to publish something to production. Here, we do some policy checks. We also do some more quality checks. Um and you might be wondering, like, we've already done some QA. Like, why are we going to do another step of QA? The reason is because we think of this like a Swiss cheese model. So, we want to try and optimize for reducing the chance of a failure getting into production. And so, there is some redundancy here or there. And that's okay. Um and so, this QA gate is is a little bit more holistic. It captures more things. But, it also will will try and flag things that we should have caught upstream, as well.