Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox

Microsoft's ThinkingBox started as a benchmark that grades agents on what they actually leave in a database rather than on what they claim. In ThinkingBox-Bench, each of 507 tasks gives an agent a real backend, a set of MCP tools and a simulated user, then runs executable checks against the final database state. This follow-up post, written jointly with Hugging Face, makes a simple observation. A check that returns 1 or 0 from the final state is also a reward function. The team used that kind of check to train Qwen3.8-27B with reinforcement learning (GRPO) for 50 steps on 187 tasks kept separate from the benchmark.

Most of the work is choosing which tasks to train on. GRPO learns by comparing several attempts at the same task. If all eight attempts fail, or all eight succeed, no attempt did better than another, and the group teaches nothing. Each attempt is also expensive here, because it needs a fresh backend, a live tool session and a separate model playing the user. So before training, the team ran each candidate task several times with the untrained model, estimated its success probability, and kept only tasks that were neither almost never nor almost always solved. They call this "offline policy-relative difficulty": a task's difficulty is measured for that particular model, once, and the training pool then stays fixed.

Each of the 507 benchmark tasks was run 20 times. Average success rose from 51.70% to 60.78%, and tasks passed on all 20 attempts rose from 38 to 107. Tasks solved at least once fell slightly, from 453 to 446. The authors read this as "sharpening": the model became more repeatable, not broader. That repeatability measure is the one the first ThinkingBox post argued headline accuracy hides. On it, the trained 27B model's 107 still trails Claude Opus 5.5's 241 and GPT-5.4's 128 in the post's comparison table, but its estimated cost per task passed every time fell from $24.36 to $8.00. The authors call that figure a comparative serving index, not a real bill, and it excludes training costs.

These are the team's own results from one reference run. The benchmark tasks were excluded from training, and their domains, databases and most tools were not seen during training. The authors note, though, that both sets share the ThinkingBox framework, simulated-user machinery, grading style and a few tools, which limits how far the transfer claim reaches. For a minority of narrow requirements the database cannot capture, the reward check also asks an LLM judge a yes-or-no rubric question. The training code is published as thinkingbox-training, but the 187-task training pool is private, and the public repository ships only sample tasks.