Stripe evals: Measuring real-world coding agent performance
- Source
- stripe.dev
- Author
- Kiran Prasad
- Date

Context
Stripe built its integration evals because, it says, coding agents lean toward older APIs that are common in their training data, miss key features and , while existing coding benchmarks only check whether code passes deterministic tests, not whether the integration is correct. It ran 69 scenarios, 24 for one-time payments, 26 for subscriptions and 19 for Connect, its product for marketplaces and platforms, on 14 pairings of a model with a coding : Claude Code, Codex or OpenCode. Each scenario ran twice with and twice without the Stripe plugin, which gives an agent Stripe-written skills and tools such as documentation search and an implementation planner through Stripe's MCP server.
The setup tries to imitate a developer who starts with a goal rather than a full spec. A second agent plays that developer: before the conversation it fixes a private list of usually two to four needs drawn from the opening prompt, then raises one unmet need per turn without writing code or revealing the list. Judges then grade each rubric criterion separately, using the visible transcript and the files the agent left behind, with each criterion marked either required or recommended. GPT 6.1 Sol served as both the roleplayer and the judge. Stripe says a check with Opus as judge showed no bias toward OpenAI models, only uniformly more lenient scoring.
The headline score is the share of criteria met, and on that measure all 14 agents improved with the plugin, by roughly 3 to 8 percentage points, while web searches fell 96%. Outdated parameters dropped from 7% to 1.3% of those used in API calls. The stricter measure, which counts a run only if every required criterion passes, is far lower. Stripe reports the average across all agents falling from 78.1% to 29.4%, which it reads as many almost-correct implementations failing the last mile. In one payments scenario, Opus 5.5 checked only that the payment had succeeded, not whether the order was still eligible to ship, which could let orders go out after a refund or dispute. Stripe also lists what the benchmark does not measure. It checks that an integration is specified correctly, not that it runs, so a well-shaped integration that does not work can score full marks. Stripe is publishing the cases and its Forge eval harness.
- Stripe found coding agents favor older APIs common in data, miss key features and hallucinate. Standard coding benchmarks miss this because they check deterministic tests, not whether the API integration is correct.
- Across 69 scenarios and 14 model and harness pairs, Stripe's agent plugin raised every pair's score by roughly 3 to 8 points, cut web searches 96%, and reduced outdated parameters from 7% to 1.3% of those in API calls.
- Requiring every MUST criterion to pass drops the average across agents from 78.1% to 29.4%. In one case Opus 5.5 checked only that a PaymentIntent succeeded, not fulfillment eligibility, which can let orders ship after a refund or dispute.
- Stripe's eval uses a roleplaying agent that derives a private set of developer needs, usually two to four and at most six, from the opening prompt and raises one unresolved need per turn, without teaching the coding agent how to pass.
- Stripe says the benchmark checks specification, not execution, so a well-shaped but non-functional integration can score full marks. It also does not measure product recommendations or how code ages with API changes.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Checking sign-in…
Loading comments…


