Vibeleaderboard
Index — Latest Intelligence

Intel

Page 14

How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp

Warp's Zach Lloyd makes the case for cloud software factories as open infrastructure defined in code.

Why it mattersFrames agent infrastructure as code with LLM-judged run scoring and per-team benchmarks, so you can compare model cost and quality on your own tasks.

videoAI Engineer

Stop Doing Agent Archaeology on Your Team's Code - Richard Feldman, Delta

Richard Feldman explains why agent-driven work loses reasoning when handed to teammates and demos Delta, where conversation, code, review and live terminal share one thread.

Why it mattersNames the cost of losing agent reasoning at handoff and shows one design: a shared thread holding conversation, diffs, review and terminal state.

videoAI Engineer

Teach Your AI Agent to Build Products Everyone Can Use - Jemima Abu, CAIS

Jemima Abu shows why coding agents copy accessibility flaws and outlines a four-step approach, including an accessibility skill.

Why it mattersGives a concrete way to make coding agents emit accessible code: define requirements, audit, use a design system, and enforce rules with a skill.

videoAI Engineer

Can AI Agents Beat Wall Street at Predicting Earnings? - Ashwin Paranjape, Samaya AI

Samaya AI's Ashwin Paranjape uses earnings prediction as an agent benchmark with a point-in-time environment, reporting frontier models beating consensus across 450+ reports and where they still fail.

Why it mattersShows how to build leak-free evaluations for financial agents and reports where frontier models now edge analyst consensus.

videoAI Engineer

What Your Coding Agent Can't See in the Code - Tom Moor, Linear

Linear's Tom Moor describes carrying context from intake to merge.

Why it mattersShows how to feed agents product history (reverts, incidents, past decisions) so reviews catch problems a standard code reviewer misses, using triage, review and lesson-extraction loops.

videoAI Engineer

AI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, Uber

Uber engineers explain how AI-driven commit volume strained their monorepo merge queue and the SubmitQueue changes.

Why it mattersShows what breaks in CI and merge infrastructure when agents multiply change volume, and the specific queue techniques that kept landing times down.

videoAI Engineer

DeepSeek Harness (DSH) EXPLAINED | Everything Is a Plugin

A walkthrough of DeepSeek Harness and its Cordis plugin system, covering dependency injection, events and cleanup, then writing a greet tool, a bash guard and a custom agent loop.

Why it mattersIt shows how DeepSeek Harness composes the agent loop, tools and system prompt as plugins, and how to write and load your own, including a custom loop that never calls a model.

videoHugging Face

NVIDIA Dynamo Recipes Bring Korea’s Leading Open Models to Production

NVIDIA published Dynamo-optimized deployment recipes for four Korean open models on B200 GPUs.

Why it mattersIf you serve A.X K2, K-EXAONE 2.0, Solar Open 2 or Motif-3, ready-made multi-GPU Dynamo recipes for B200 cut the routing and parallelism tuning work. NVIDIA claims up to 2x throughput out of the box.

blogAmr Elmeleegy

Guardrail Erosion in Domain Fine-Tuned Large Language Models: A Paired Audit of Knowledge-Centric Abstention in Biomedical LLMs

An audit of 25 biomedical fine-tuned LLMs against 11 paired base checkpoints on MedAbstain and ClinDet-Bench.

Why it mattersIf you ship domain fine-tuned models, this audit finds appropriate abstention was already low in base models (median 8.3% on MedAbstain) and dropped further after fine-tuning, so test it before deployment.

articleStephen Peng, Vineet Jagadeesan Nair, Noah Broestl, Anne Friederike Kleppe, Steven Mills, Adi Zolotov

Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine

Overload experiments on vLLM show cancellation propagation did not restore service with retries on.

Why it mattersIf you run vLLM behind retrying clients, propagating cancellation will not restore service after overload. Retries waste prefill work and metrics undercount it, so you need retry budgets or request coalescing.

articleXindan Zhang, Fei Wang

Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$

Compares a pdb-only agent with a bash-only agent across all 500 SWE-bench Verified tasks on two open models, holding scaffolding fixed.

Why it mattersIsolates whether giving a coding agent an interactive debugger instead of shell loops changes what it can solve, and offers a metric to separate interface effects from extra sampling coverage.

articleRyan Magnuson, Ellie Kitanidis, Ariel Lee

Decision Context: Persistent Behavioral Learning for AI Agents

Forge Decision Context turns human corrections and tool-call failures into imperative rules injected where they apply.

Why it mattersSuggests procedural corrections work better as explicit imperative rules placed at the decision point than as retrieved memory, cutting injected tokens roughly tenfold in the authors' benchmark.

articleNima Vahdat, Pashootan Vaezipoor

Decision models take 26% of requests; Anthropic now wins new teams 2:1

Vercel's monthly AI Gateway index reports decision models at 26% of requests but under 1% of spend, voice model adoption rising, Anthropic winning new teams nearly 2:1.

Why it mattersProduction data shows decision models at 26% of requests but about half a cent per dollar, and that most teams move off a model within a month of its peak. Useful for routing and model-selection decisions.

articleAmelia Charles

Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP

Tuned-lens and activation analysis on Llama-3.1-8B and Qwen3-8B shows models commit to an answer letter early under MCQ format, sitting about 3x further from the refusal direction.

Why it mattersMCQ safety benchmarks may mostly measure knowledge rather than safety, since forcing an answer letter bypasses refusal. Treat MCQ-based safety scores with caution when evaluating models.

articleStepan Epifantsev, Anna Antipova, Oleg Chernikov, Leon Strelkov, Alexey Dontsov

MongoBench: Do LLMs Know What a MongoDB Query Will Do?

MongoBench tests whether LLMs can predict the exact documents a MongoDB find() filter returns, with ground truth from a real server.

Why it mattersProducing a valid query does not mean a model knows what it will match. Agents operating live databases need execution-grounded checks, since even frontier models misjudge their own tricky queries.

articleQibai Chen, Haoting Qiu, Yezhou Cheng, Mengxiao Qian

When Routine Operations Break Architectural Pattern Relationships in LLM Agent Harnesses

Analysis of five agent harnesses and fourteen plugins found routine operations broke guardrail pattern relationships without code changes.

Why it mattersIf you rely on plugin-based guardrails in an agent harness, routine operations can silently disable policy evaluation while plugins appear active. Verify enforcement order and state after config changes.

articleBoming Xia, Qinghua Lu, Dino Sejdinovic, Liming Zhu

Hide your slop PR descriptions behind

Simon Willison's workflow for PRs opened by Claude Code and Codex.

Why it mattersAgent-written PR descriptions are accurate but long enough that reviewers skip them. Keeping a short human summary visible and collapsing the rest preserves the detail without costing reviewer attention.

articlesimonwillison.net

Engineering Agency out of the Happy Path — Matthew Jewkes, Standard Cybernetics

A Clarivate case study on lifting a biopharma indexing automation from roughly 80% to 99.95% accuracy.

Why it mattersShows how splitting model errors, missing organizational answers and genuinely new cases into separate routes lets a routine path reach 99.95% accuracy while escalating the rest, a reusable pattern for high-reliability agent systems.

videoAI Engineer

I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft

A product manager recounts three weeks of agent-led refactoring that left tests calling production inference and an 'agent' that was still a workflow.

Why it mattersShows concrete failure modes of unsupervised coding-agent refactors (tests hitting production inference, labels that don't match behavior) and a workflow of owning goals and evaluation rubrics while delegating implementation.

videoAI Engineer

Profile Relevance Boosting Feed Relevance With Llms At Linkedin

LinkedIn engineers describe Profile Relevance, an unsupervised retrieval system using LLM-generated embeddings to match members with posts from outside their network.

Why it mattersShows how LLM embeddings of professional context can retrieve unconnected content in a large production feed without supervised labels. Useful as a reference for embedding-based retrieval design.

articleLinkedIn Engineering

Investigate Kubernetes health and processes with the Datadog MCP Server

Datadog expands its MCP Server with Kubernetes control plane health tools, Live Processes search and a processes skill that guides agents to the right telemetry source for infrastructure investigations.

Why it mattersAgents investigating Kubernetes incidents can now check control plane health in one call and search process-level data through MCP, with a skill that routes each question to the right source.

articlewww.datadoghq.com

Endpoint Accuracy Index V1 1

Artificial Analysis updates its Endpoint Accuracy Index to v1.1 with 25 repeats on long context, tighter intervals and new open-weights coverage.

Why it mattersProviders quantize, cap outputs and ship bugs, so the same model can score differently by endpoint. This index shows which endpoints fall below a self-hosted reference on tool calling, reasoning and long context.

articleArtificial Analysis articles

Claude Fable 5.2 now available on AI Gateway

Vercel AI Gateway now serves Claude Fable 5.2.

Why it mattersIf your team uses zero data retention, requests to Fable 5.2 through AI Gateway will fail. Add a models fallback array so classifier declines fall through to another model.

articleWalter Korman

Meta S Muse Ember Is The Cheapest Model Scoring 36 On The Intelligence Index

Artificial Analysis benchmarks Meta's Muse Ember.

Why it mattersMeta's Muse Ember matches Claude Haiku 5.5 on the Intelligence Index at about a third of the cost per task ($0.14), but with weak knowledge accuracy. This helps you choose a cheap model for agentic workloads.

articleArtificial Analysis articles

Workflow SDK now writes over WebSockets on Vercel

Workflow SDK 5.2.0 sends event and stream writes to Vercel Workflows over WebSocket by default.

Why it mattersUpgrading to workflow 5.2.0 makes step events and AI token streams use a WebSocket with no code changes. A replayed agent stream finished 4.2 times faster than v4.8.12.

articleAlex Langenfeld
Artificial Analysis@ArtificialAnlys

Artificial Analysis releases Decision Index v2: a calibration-aware Decision Score, private AA-Paperclip and AA-Symmetry benchmarks, and 45% private data. Decision models lead on cost and latency, while Claude Haiku 5.5 and GPT-6 Luna score about 3 points higher at far greater cost.

Why it mattersDecision Index v2 scores correctness and calibrated confidence together, with 45% private data against gaming. Cost per 1M tasks varies over 60x across decision models, and billing per question versus per document is a major cost driver.

open-webui v0.13.3

Open WebUI 0.13.3 adds coding agents to automations. You pick a terminal and folder, scheduled runs verify both are still available, and results post to a chat.

Why it mattersOpen WebUI scheduled automations can now launch a coding agent in a saved terminal and working folder, with results sent to a chat. Scheduled agent runs no longer need a separate scheduler.

repogithub-actions[bot]

parse_number() as a SQLite extension: C vs Rust

Simon Willison had Claude build four parse_number() SQLite extensions in C and Rust, then compared size and speed.

Why it mattersGives measured numbers: optimized C ran 16 to 19 ns per row, about twice as fast as Rust, with a 14 KB binary versus roughly 300 KB. It also shows an agent-driven benchmark workflow backed by fuzz-testing.

How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco

Mihail Eric walks through Stanford's CS146S syllabus, where students propose patches to real open-source repos for maintainers to review.

Why it mattersShows how a Stanford course restructured assignments once agents can one-shot homework, and which agent tooling a real team chose to build or buy. Useful for deciding what engineers should learn and own.

articleswyx .ai

What the Top Companies Do Differently With AI Coding - Martin Harrysson & Prakhar Dixit, McKinsey

McKinsey reports that roughly the top quarter of large firms see 2x gains from agentic coding, citing shorter cycles, smaller merged-role pods, curated skills.

Why it mattersDescribes what the top 20 to 25% of enterprises do differently with coding agents and where the bottleneck shifts once generation speeds up, which helps teams decide where to invest beyond code generation.

videoAI Engineer

crewAI 1.15.38

crewAI 1.15.38 makes gpt-6-luna the default model, reads spreadsheet URLs in agent input, adds --no-open for headless crewai eval, tags eval usage spans.

Why it matterscrewAI 1.15.38 changes the default model everywhere, adds a headless flag for crewai eval, and makes failed project scripts exit non-zero, which affects CI and unattended runs.

repojoaomdmoura

How I Fight AI Brain Fry

A practitioner's method for avoiding context overload with terminal coding agents.

Why it mattersOffers a concrete way to tame parallel Claude Code or Codex sessions: keep state in git-committed tickets, let the agent work a queue, and review only decisions in one prioritized inbox.

How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp

Ramp's applied science lead explains evals for the Stack bookkeeping agent.

Why it mattersShows how to build repeatable agent evals with synthetic data and back-tests, and which experiments helped (skill ablation, memory tests, scheduled regressions) versus failed (auto-tuning prompts).

videoAI Engineer

Goldman Sachs Is Pulling Claude Code from Its Bankers and Traders

Goldman Sachs updated internal guidance in October to limit Claude Code and GitHub tooling to engineers, ending use by traders and bankers who built flow-tracking and market-model tools.

Why it mattersShows a major bank pulling coding agents from non-engineers over risk and productivity concerns. Teams rolling out agent tooling to non-technical staff can see where governance pushback is landing.

Claude Code Looked Right but Missed $800K - Natalia Connolly & Kevin Madura, AlixPartners

AlixPartners used Claude Code on 40,000+ contracts and found plausible answers off by roughly $800K.

Why it mattersLists concrete silent failure modes in agentic data analysis, such as access errors that look like zeros and orphaned amendments, and the structured pipeline that made answers traceable to sources.

videoAI Engineer
Lenny Rachitsky

What it actually takes to build frontier models | Tulsee Doshi (Head of Product for Gemini)

podcast

open-webui v0.13.1

Open WebUI 0.13.1 adds coding agents (Claude Code, Codex, Pi) as selectable models on compatible Open Terminals, with reasoning-effort controls and in-chat tool approvals.

Why it mattersOpen WebUI can now run coding agents like Claude Code and Codex from its model picker, with tool-request approvals in chat. Tether links your own computer so agents work in your real repositories.

repogithub-actions[bot]

How Financial Planners With Zero Coding Experience Built Their Own Agents - Matt Ragone, Range

A financial planner at Range explains how non-technical planners built their own agents and skills.

Why it mattersShows a repeatable five-step process for turning a recurring workflow into a tested agent skill with a named owner and reviewer. Useful for anyone rolling out agents to domain experts who do not code.

videoAI Engineer

Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah

Tanmay and Dolly Sah present UndoBench and EvoUndo, showing coding agents can't reliably reverse their own actions. About 42% of capability-improving changes broke recovery.

Why it mattersGit does not reverse external side effects, so agents need captured prior state and a chosen recovery strategy.

videoAI Engineer

Your Company Has No Model of Itself. Agents Can Build One - Joao Wenzel, Citadel Securities

Citadel Securities' Joao Wenzel shows an agentic institutional memory modeled on robotic state estimation.

Why it mattersDescribes an update-age-observe-grade-correct loop for agent memory, where facts decay and predictions are back-tested against outcomes. Gives builders a design beyond plain retrieval for long-running organizational agents.

videoAI Engineer

Goodbye Claude, Hello Pi

A longtime Claude Code user explains why they moved to the Pi harness.

Why it mattersNames specific Claude Code pain points (IDE session limits, missing AGENTS.md support, extension gaps) that matter when choosing a coding harness. Useful if you are weighing a switch to a minimal harness like Pi.

GitHub’s Growth: 225M Developers, 34.3B Commits — Jay Parikh, EVP of CoreAI, Microsoft

Microsoft CoreAI's Jay Parikh discusses agent-driven load on GitHub, Copilot Fusion model orchestration, session portability across devices, the Azure move.

Why it mattersExplains how GitHub is rebuilding its platform for agent traffic, including model orchestration for cost versus quality, which affects how engineers plan agent workflows and spend.

videoLatent Space

Decision Index V2

Artificial Analysis releases Decision Index v2.

Why it mattersThe index now scores both correctness and calibrated confidence, and 45% of it is private data, which limits benchmark gaming. Cost per 1M tasks varies over 60x across models, partly because of how document input is billed.

articleArtificial Analysis articles

Don't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal

Modal's CTO lays out what agents need from infrastructure.

Why it mattersAgent builders get a checklist of infrastructure needs: persistent sandboxes without idle cost, elastic resources, network controls that stop credential leaks, and observability agents can read.

videoAI Engineer

We Visited China’s Stargate in Inner Mongolia

SemiAnalysis tours Galaxy, a planned 800MW datacenter campus in Ulanqab, Inner Mongolia.

Why it mattersA 120MW phase delivered within a year at a fifth of US cost shows how fast China can add AI compute. That informs how you reason about future capacity, inference pricing and availability.

articleEverlyn Tan

Build Your Own Finance Agent Inside ChatGPT - Stefano Fabbri & Oliver Steffmann, OpenAI

OpenAI's applied AI team walks through building financial workflows in ChatGPT and Codex.

Why it mattersShows how to package skills and MCP connections into plugins, and extend them with Codex on internal data. Covers when to use skills versus plugins and how Codex sandboxing works.

videoAI Engineer

The end of Microsoft Research's AI Frontiers group

John Langford recounts how Microsoft Research's roughly 50-person AI Frontiers group, behind AutoGen, the Fara computer-use agent and new optimizer and architecture work, was folded into Microsoft AI.

Why it mattersA researcher's firsthand account of a frontier lab choosing scaling over research, which affects which work (agent frameworks, optimizers, architectures) continues.

Your Financial Agent Needs to Show Its Work - Anirudh Veeraragavan, Fieldguide

A Fieldguide security engineer presents four controls for auditable financial agents.

Why it mattersGives a four-part control set for agents in audited domains: block overwriting human-reviewed work, make steps replayable, pin citations to document versions, and audit eval data so a 97% score does not hide failures.

videoAI Engineer

MaxxBench – "Nice DeepSWE score. But is it true?"

MaxxBench alters instructions or starter code in 89 DeepSWE tasks.

Why it mattersSmall requirement changes to familiar DeepSWE tasks cause large score drops, because models guess what hidden tests expect. A high benchmark score may not mean an agent follows your actual spec.

articleantiuniverse

How To Build The Future: Paul Graham & Amjad Masad

Paul Graham interviews Replit CEO Amjad Masad on building an early coding agent, why enterprises are moving projects onto Replit, what comes after vibe coding.

Why it mattersReplit's CEO describes how its 2024 agent worked through errors and provisioned infrastructure, and how enterprises are replacing large software projects with agent-built ones. It shows where coding agents are heading commercially.

videoY Combinator

An index of the vibe-coding frontier. Corrections welcome.