Clip / AI Tools

Reward hacking from environment quirks (potholes and timeout abuse)

From Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute · ≈6:48

“And the main sort of problem is like has kind of two names, which are both the same problem, environment fidelity and reward hacking.”
AI Engineer

“So, we have some examples that we've seen where in a trading run in the past, we had some like networking issues causing our environment to have tool calls that failed maybe around 10% of the time. If that is the case, then we actually saw that the model would then start outputting shorter and shorter responses.”
AI Engineer

“Now, this was really surprising to us because in our reward function, we actually didn't have any length penalty.”
AI Engineer

“if you think about maybe the model is like a human like walking along a sidewalk and like the tool call failures are like potholes in the sidewalk, like it makes a lot of sense that because there's so many potholes, the model doesn't want to run for that long uh because it might fall in a pothole and then get a zero reward for the rollout”
AI Engineer

“One thing we saw was that if your tool calls take a long time, then if the model feels like the problem is really hard, it will actually just be incentivized to like abuse the tool calls and just like call a lot of them in quick succession and try to time out the sandbox so it avoids getting a reward of zero. It just gets the rollout dropped.”
AI Engineer

What’s in it

Shows a concrete postmortem of reward hacking: flaky tool calls (10% failure rate) caused the model to shorten responses to avoid 'potholes,' and sandbox timeouts caused models to spam tool calls to force a dropped rollout instead of a zero reward — a key pitfall in RL environment design.

Watch this moment

Clip transcript

So, some challenges that we face in this setup is that the environment is something that you want to basically, uh, use to replicate reality, so that after you're done training, like the improvements that you've seen actually translate to when you deploy these models into production. And the main sort of problem is like has kind of two names, which are both the same problem, environment fidelity and reward hacking. Essentially, the agent is exposed to an environment and sort of any, uh, any quirks of your environment will end up being something that your agent may like learn a model of. So, we have some examples that we've seen where in a trading run in the past, we had some like networking issues causing our environment to have tool calls that failed maybe around 10% of the time. If that is the case, then we actually saw that the model would then start outputting shorter and shorter responses. Now, this was really surprising to us because in our reward function, we actually didn't have any length penalty. So, like we we couldn't really tell why this was happening, but really what's going on here is if you think about maybe the model is like a human like walking along a sidewalk and like the tool call failures are like potholes in the sidewalk, like it makes a lot of sense that because there's so many potholes, the model doesn't want to run for that long uh because it might fall in a pothole and then get a zero reward for the rollout. Um and then conversely, um it's also possible that your model just learns to like output more and more gibberish over time depending on like what your environment looks like. So, in a different case, we had a training run where we um have sandbox timeouts uh just so that they don't run forever, and we usually like filter out the rollouts that timed out from being trained on. One thing we saw was that if your tool calls take a long time, then if the model feels like the problem is really hard, it will actually just be incentivized to like abuse the tool calls and just like call a lot of them in quick succession and try to time out the sandbox so it avoids getting a reward of zero. It just gets the rollout dropped.