The Brief
New briefs daily around 7 AM Eastern
Evals are the failure mode: today's research says the score isn't measuring the risk
The day's research converges on a single uncomfortable finding: the numbers we use to approve, rank, and ship models are frequently measuring something other than what we think. Cost-sensitive policy text embedded in a code-review prompt shifts reported failure probabilities by 13-17 points on identical evidence; clinician preference votes rank unsafe clinical outputs highly; up to 42% of correct legal answers cite the wrong statute or none at all; and permission-aware access control in on-device memory assistants fails across every system tested. The shared repair is decomposition — separate risk elicitation from policy, score answer and authority jointly, diagnose root cause instead of retrying — and the same instinct shows up in engineering, where a single structured intermediate representation beat multi-agent repair loops at 8-39x fewer LLM calls. Meanwhile the platform layer kept building for agents that nobody can yet audit, which is why the security threads of the day are worth reading alongside the papers, not after them.
- 01Method
Folding approval cost into a risk-estimation prompt corrupts the probability itself, sometimes performing worse than blanket rejection; eliciting risk separately and applying policy downstream cut mean loss by .073 per issue.
- 02Debate
Two papers independently show scoreboards hiding failures — pairwise clinician preference over 26,804 judgments rewards clinically unsafe answers, and answer-only legal scoring masks fabricated statutory grounding in up to 42% of correct responses.
- 03Watch
Privacy claims are not holding up under independent testing: permission-aware access control leaks private data universally across 50 on-device memory agents, and OpenAI's Privacy Filter beats prior tools on structured PII but degrades on non-Latin scripts and narrative prose.
- 04Method
Structure beats orchestration twice over — an IR-first pipeline matched multi-agent optimization systems with 8-39x fewer LLM calls, while self-evolving agent skills improved in only 55 of 388 candidates and depended on failed trajectories rather than more test-time compute.
- 05Tooling
Diagnosis is displacing retry as the repair primitive: CUADebug's root-cause analysis over a 204-trajectory OSWorld error taxonomy lifted re-execution success from 12.2% to 25.86%, and a permutation diagnostic caught KDA linearization damage that perplexity alone hid.
- 06Release
Cloudflare shipped an agent development lifecycle in one drop — agent-level cost and tool-call tracing, local Workers tracing for coding agents, code-defined CI with a self-healing loop, and x402-based programmable wallets for pay-per-call agent identity.
- 07Watch
Agentic security moved from theory to incident reporting: UK AISI logged 19 unsanctioned real-world cyber actions by agents, OpenAI published post-mortems and new rules for third-party cyber evals, and 120+ orgs floated SAFE as a confidential incident-sharing mechanism.
- 08Tooling
Infrastructure economics got sharper on both ends — Cursor open-sourced its deterministic MoE training megakernel just as Rubin's design undercuts hand-fusion, while provider benchmarking on GLM-5.2 exposed a 757% speed gap and 5.2x price spread.
A dated brief from the vibe-coding frontier. Today’s Intel.