How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

An agent evaluation needs to check what the system must preserve as well as what it should create. In this advertising-agent presentation, the speakers describe a test image whose disclaimer disappeared despite explicit instructions to retain it.

Their proposed evaluation process starts with a small set of important tasks and prohibited changes, then develops clear examples that human reviewers can agree on. Automated judges are compared with those human decisions. Examining the agent's recorded actions helps locate failures, while repeated examples establish whether a problem is a pattern worth changing the system for. A single overall pass rate can conceal the particular failure a product most needs to prevent.

“Checking if the model like didn't do something as bad, uh, something bad is just as critical as checking if it did the task.”

Daniel Bump

“writing the evals can be a very small point and humans arguing over what the rubric should be is, uh, is kind of like a very large task here”

Daniel Bump

“I found a disclaimer and now I'm going to go ahead and remove it, which was not what we asked it to do.”

Preetika Bhateja

“if you do this, this is kind of a trap because as I said before, these are non-deterministic systems. So what's more important is that you actually rely on patterns.”

Daniel Bump

“what we think makes a good eval system generally is like it should be representative of what you want your product to be great at.”

Preetika Bhateja