From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

A record of an agent's failure becomes more useful when engineers can reproduce the situation and test a fix. Snorkel AI's simulation talk describes turning execution traces into repeatable tasks with the relevant files, database state and tools. Different agent configurations can then face the same environment, making comparisons of success, cost and time more meaningful.

The test environment needs its own checks. A known working solution establishes that the task is solvable, while separate verifiers inspect the resulting state and outputs. The speaker also recommends checking that doing nothing fails. These checks help distinguish a broken agent from an impossible task or a test that awards success too easily.

“Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship.”

Rustem Feyzkhanov

“because in your production, you don't care about the model, you care about the full system.”

Rustem Feyzkhanov

“Agent can try to reward hack simulation environment because it can understand that it's in simulation and it can hack it.”

Rustem Feyzkhanov

“effectively benchmark is software. It's code. It's files. You need to treat it as such. You need to have a separate CI pipeline for it.”

Rustem Feyzkhanov

“with simulation, you control the full stack. You can evaluate the full stack. And you can make sure that fix lives in the correct place.”

Rustem Feyzkhanov