The Brief
New briefs daily around 7 AM Eastern
The evidence problem: today's research says passing tests prove less than you think
A cluster of papers landing today converges on one uncomfortable finding: the signals we use to judge AI-written code are weaker than the code itself. Repair agents cite passing tests that nearly half the time cannot distinguish a real fix from a no-op, coding models quietly guard dead code instead of deleting it so suites stay green, and a production case study puts the hidden tax in numbers — 2.3 fixes for every feature shipped. The mitigations arriving alongside these diagnoses are notably unglamorous and mostly training-free: gate the agent until it has gathered evidence, replay patches against the buggy version, stage a migration instead of one-shotting it, audit judges with a second model chosen per bias type. The same skepticism is spreading to evaluation itself, where fine-tuning benchmarks, difficulty ratings, and leaderboard scores are all being shown to rank things they don't actually measure.
- 01Method
ECLoop adds a model-agnostic execution layer that blocks coding agents from editing before they have gathered sufficient evidence, lifting SWE-bench Verified Pass@1 by 4.8–11.8 points and cutting tokens up to 12.1% with no retraining.
- 02Debate
Two papers attack the same blind spot from opposite ends: repair agents' passing tests fail to discriminate a correct fix roughly half the time, while code editors avoid deletion by wrapping dead code in guards — both produce green suites and worse codebases.
- 03Watch
Prompt wording is not neutral: the MASC framework shows benign metaphors like 'search the crowd' steer models toward less efficient algorithms, with the shift detectable in hidden states.
- 04People
An experience report from building PM4Py-UCM entirely with Claude Code quantifies the overhead honestly — 374 human turns, 10K+ tool actions, and fixes outnumbering features 2.3:1 — and pairs it with oracle-based validation to catch agent errors early.
- 05Tooling
Testing work is moving from selectors to intent, with DragonCrawl using GPT-4o multimodal reasoning to cut mobile E2E onboarding from 96–120 hours to under four, and LLM-generated metamorphic relations producing 7,004 oracles across 142 AR repositories.
- 06Method
A Ship-of-Theseus workflow for C-to-Rust migration argues for a safe-but-ugly baseline followed by incremental idiomatic rewrites validated by compilation and behavioural tests, demonstrated on a 12.5k SLOC codebase.
- 07Debate
Evaluation itself is under audit — a talk on benchmark gaming, evidence that GLUE fine-tuning misranks federated pre-training runs, GPT-5 losing calibration on hard items where a small ConvBERT wins, and Chain-of-Models showing per-bias auditor selection beats picking one 'unbiased' judge (0.884 vs 0.824/0.805).
- 08Release
Open-weight momentum continues with Laguna S2.1, Inkling and Kimi K3 pushing the Pareto frontier and licensing terms diverging sharply — Apache 2.0 and OpenMDW on one side, noncommercial revenue-share on the other — while multimodal work flags a measurable accuracy drop when identical questions arrive as images rather than text.
A dated brief from the vibe-coding frontier. Today’s Intel.