The Brief
New briefs daily around 7 AM Eastern
Four papers land today against LLM-as-judge, and for executable checks
Four papers published today attack the same assumption from different directions: that a model's assessment of an output, its own or another model's, is reliable enough to label data or gate a loop. Function-level execution feedback compared judge annotations against actually running the code and found the judge marking working functions as failures, corrupting the preference data trained on them, while a separate study shows multi-dimensional judge rubrics leak across dimensions rather than scoring independently. The termination paper carries the point into agent design, replacing a model's self-declared doneness with a replayable evidence certificate it must produce before COMPLETE. If you label, score or gate with a judge, the cheap correction is not a better rubric, it is an executable check the judge cannot talk its way past.
- 01Read
Function-level execution feedback compared judge annotations against actually running the code, and found the judge marking working functions as failures, which poisons the preference data trained on them. A separate paper shows multi-dimensional judge rubrics are not independent, with one dimension's score leaking into another, and offers step-wise chain-of-thought pruning that reduces the leakage across models and tasks. A third reports near-zero correlation between automatic creativity metrics and human judgment, plus a systematic pull toward AI-styled text. Where the artifact can be run, run it.
- 02Read
Self-judged completion breaks under exactly the conditions agent loops produce, lost tool responses and partial commits, where the model reports done over a half-applied change. The proposal is a replayable evidence certificate the agent must carry before it may emit COMPLETE, which is a gate you can bolt onto an existing loop without retraining anything.
- 03Read
Qwen3.8-Flash-Next ships open weights for a multimodal MoE with roughly 6B active parameters out of 125B, priced near $0.16 and $0.47 per million tokens, which the lab frames as an early look at Qwen4. SemiAnalysis itemizes what is inside: a 51-billion-parameter n-gram embedding table cheap enough to push onto slower memory tiers, gated residual connections, and a lightning indexer selecting context at micro-block granularity. The architecture can be read rather than inferred, which is the part worth your afternoon.
- 04Read
ToolRobustBench perturbs interface, intent, output and runtime as separate stages, and reports tool-calling accuracy degrading sharply and unevenly, with stage attribution naming which part of the harness gave way. ESQ-Bench finds the enterprise version of the same gap, NL2SQL accuracy above 89% on academic sets falling to between 57% and 80% on tiered Oracle schemas, and its silent-divergence metric catches the worst case: queries that execute cleanly and return the wrong rows.
- 05Try
Codex rust-v0.150.0 suppresses AGENTS.md in untrusted projects, so a cloned repository can no longer hand instructions to the agent reading it. Cline v4.1.16 redacts credentials embedded in git remote URLs before workspace information reaches the model. Two unrelated tools shipped the same premise the same day: the working directory is model input, and input from strangers gets sanitized first.
- 06Read
Anthropic opened real Claude and Claude Code conversation data to independent researchers, and the first result reports that over half of sampled conversations involved consequential, hard-to-undo work. That figure is the case for the gates the papers above describe, since it measures how much everyday agent use already carries irreversible side effects. A METR productivity study is queued behind it.
- 07Read
Cowork's desktop app gains an Anthropic-controlled browser in a side panel that navigates and fills forms inside a session held separate from the user's own profile and logins. The separation is the whole feature, since agentic browsing on a live cookie jar is the failure it avoids. Rolling out to paid plans over the next week, with Claude in Chrome now generally available.
A dated brief from the vibe-coding frontier. Today’s Intel.