Vibeleaderboard
Index — Latest Intelligence

Intel

Page 24
Artificial Analysis@ArtificialAnlys

GitHub's Copilot Fusion orchestrates Claude and GPT models at runtime. Artificial Analysis benchmarked its max setting: $5.59 per task versus $11.42 for Claude Code on Opus 5.5, about half the tokens, a Terminal-Bench 4.0 lead, and weaker SWE-Atlas-QnA results.

Why it mattersCopilot Fusion routes between Anthropic and OpenAI models and, per Artificial Analysis, scores 2 points below Claude Code with Opus 5.5 at about half the cost per task.

GitHub Universe 2026: Evolving every layer of the stack

At Universe 2026 GitHub laid out an agentic software delivery system built on velocity, intelligence and quality.

Why it mattersGitHub is reframing its platform around orchestrating fleets of agents to produce completed work, which affects how teams using Copilot and Actions will plan, review and ship.

blogJay Parikh

Octoverse: Commits, pushes, reach new highs in 2026 as developers build with agents

GitHub's 2026 Octoverse reports 17.9 billion pushes in 12 months and finds new repos with an agent configuration file logging eight times the commits of those without.

Why it mattersGitHub's own data shows agent use correlating with sharply higher commit and PR volume, giving engineers hard numbers on how agentic workflows are changing software output.

blogGitHub Staff

Dualchem Evaluating Ai Models Dual Use Chemistry Risks

Scale AI introduces DualChem, a 600-question benchmark of ambiguous dual-use chemistry requests built with NIST CAISSI.

Why it mattersGives a concrete benchmark for how frontier models handle ambiguous dual-use requests. Relevant if you build on or evaluate models where safeguards affect permitted use.

articleScale AI blog

Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

Surge AI's DAYJOB: Tax benchmarks agents on 50 professional tax assignments with incomplete instructions and conflicting records, scoring ownership, initiative and judgment.

Why it mattersIt tests whether agents surface issues a client did not ask about, not just compute correctly. Results show models differing on that initiative, which matters when delegating professional work to agents.

articlelandonxi
Artificial Analysis@ArtificialAnlys

Artificial Analysis benchmarks NVIDIA's Nemotron 3.5 Transcribe, a 30B total, 4B active open-weight speech-to-text model. It ties for first among open models at 2.8% WER, leads on voice agent conversations, and runs at 75x real time on DeepInfra for $2 per 1,000 minutes. Only the opening of the thre

Why it mattersAn open-weight transcription model matching a 24B-active competitor with 4B active parameters lowers the serving cost of voice agent pipelines. Hosted pricing of $2 per 1,000 minutes gives a concrete reference for build decisions.

Thin, Light and Built for Everything: NVIDIA RTX Spark Windows Laptops Gain New Models, Agent Tools and Games

NVIDIA says RTX Spark Windows laptops are available with up to 128GB unified memory and 1 petaflop FP4.

Why it mattersLocal inference hardware with 128GB unified memory in a laptop changes which models you can run on-device. The new GPU split between foreground apps and background agents targets local agent workloads.

blogShreevar Rastogi

NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt

NVIDIA released Nemotron 3.5 Transcribe, an open-weight model for multilingual transcription, diarization, translation and audio Q&A.

Why it mattersOne open checkpoint now handles transcription, speaker labels and translation, steerable by prompt and adaptable with released data and recipes. Useful for voice features without a hosted API.

articlehuggingface.co

unsloth v0.1.912-beta — Trick or Treat + Optimizations

Unsloth v0.1.912-beta batches up to four compatible fine-tuning jobs with multi-LoRA, halves first-launch time, cuts Linux tool-call startup to 1 to 2 ms, extends thinking and auto-compact to SGLang.

Why it mattersMulti-LoRA batching trains up to four compatible queued fine-tuning jobs together, and the measured load and tool-call speedups cut iteration time for local fine-tuning.

repodanielhanchen
MistralAI@MistralAI

Mistral opens its Responses API in public preview for agent building. Each response splits the answer, reasoning and tool calls into separate items, and it works with every Mistral-hosted chat model including Large 4. Also adds DeepSeek V4.1 Flash and GLM 5.3 Flash.

Why it mattersMistral's Responses API returns the answer, reasoning and tool calls as separate items, so agent code can handle each step without parsing a single Chat Completions message. It works with all Mistral-hosted chat models.

The October inflection point for vibe game development

Simon Willison argues that October 2026 is an inflection point for vibe coding games.

Why it mattersShows concretely what current models can do for game development without code review, including a simple prompting pattern that improved gameplay. Helps you judge where agent-built games are now viable.

articlesimonwillison.net

llama.cpp v0.8.0

llama.cpp v0.8.0 adds LiquidAI's LFM3 text, vision, audio and omni models, an experimental streaming-audio API, and server endpoints for realtime audio and TTS.

Why it mattersLocal inference now gets realtime audio in and out through the server plus several new model architectures, so engineers running llama.cpp can build voice and omni workflows without a separate stack.

repogithub-actions[bot]

pplx embed v2 context 9b released

Perplexity ships the 9B version of its contextual multimodal embedding family under an MIT license.

Why it mattersAn MIT-licensed 9B multimodal embedder whose 1024-dim prefix is compatible with the 0.6B model lets you mix a large indexing model with a cheap query model. Contextual chunking also applies to images.

articlePerplexity model releases

Liquid LFM3 Is INSANE – Tiny Local AI That ANYONE Can Run!

A hands-on evaluation of Liquid AI's LFM3 small-model family.

Why it mattersShows how small on-device LFM3 models handle agent tool workflows, vision, and voice, including on phones. Useful for deciding whether local models are viable for edge agents.

videoBijan Bowen

Can AI Get Immigration Right?

Vals AI's Immigration Bench scores 25 models on expert rubrics.

Why it mattersEven the top model fully answers only about a quarter of immigration questions, and recent rule changes are nearly invisible to every model. Teams building legal or advice products should expect stale-knowledge failures and add retrieval.

articlewww.vals.ai

Vercel Container Registry now supports 2 GB compressed image layers

Vercel Container Registry raises the compressed layer limit from 500 MB to 2 GB while keeping the 15 GB total image cap.

Why it mattersLarge toolchains and dependency sets, such as custom sandbox images for coding-agent evals, can now be pushed without splitting layers. The per-layer limit is 2 GB.

articleAndy Waller
Evan Fenster@EvanFenster

Channel3 reports on using TypeSafe AI's Jev decision model for product categorization. Beam search over the taxonomy failed, but reranking 30 embedding candidates reached 97.2% accuracy versus 81.9%, at similar cost. It also notes cases where cheap batched models match it.

Why it mattersClassifying against a large taxonomy is a common agent pipeline step.

Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod

Runpod's Chris Vela demonstrates data-driven vLLM configuration with a live benchmarking dashboard, version and CUDA pitfalls, batch-token tuning for latency.

Why it mattersShows how to pick vLLM settings from measured concurrency and latency instead of defaults, including how max batch tokens affects inter-token latency. Helps teams self-hosting open models.

videoAI Engineer

Who Is Making This Request? Agent Auth 101 - Kamil Potrec, Keycard

Keycard's Kamil Potrec explains why current agent credential practices fail and what proper agent auth needs, then compares SDK and gateway approaches and their tradeoffs.

Why it mattersStatic keys and personal tokens make it impossible to tell which agent made a request. The talk lays out per-agent identity, scoped short-lived credentials, and gateway designs that keep secrets out of agent reach.

videoAI Engineer

Introducing Virtual Models on AI Gateway

Vercel explains virtual models on AI Gateway.

Why it mattersWith new models arriving weekly, moving model choice out of code lets you change models, fallbacks and data-retention policy for agents without a release cycle.

articleRohan Taneja

Exploits Now Beat Patches: Open Source Security in the AI Era - Patrick Smyth, Chainguard

Chainguard's Patrick Smyth covers AI-assisted attacks and supply chain malware waves, four attack surfaces including agent skills.

Why it mattersArgues that 30-day remediation windows fail now that exploits arrive before patches, and lists actions: cooldown periods, signing with SLSA and Sigstore, and securing CI/CD, laptops, and agent skills.

videoAI Engineer

Introducing Vercel CI

Vercel CI enters beta, running only the Turborepo tasks affected by a change on per-task compute with Remote Cache on by default.

Why it mattersWhen coding agents push many PRs, running only the tasks affected by a change cuts CI wait. Vercel reports 2-6x faster push-to-pass internally, configured by tagging tasks in turbo.json.

articleAnthony Shew

Continue or Restart? The Limits of Early Failure Prediction in Coding Agents

A study of 147,002 coding-agent traces finds early execution prefixes barely improve prediction of failure within a task.

Why it mattersKilling a coding agent run early to save compute sounds appealing, but early trajectory signals do not separate good from bad attempts on the same task. Restart-or-continue heuristics based on early prefixes are unreliable.

articleSrinath Srinivasan, Lunxiao Li, Tim Menzies, Wesley Assun\c{c}\~ao

PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

PROBE benchmarks frontier coding agents on 2,457 bugs fixed over 18 months in 20 GitHub repositories.

Why it mattersCoding agents asked to find bugs across a repository recover under 21% of what maintainers fixed, and stronger models raise recall but not precision. Their findings are mostly valid yet skew toward different bugs than humans fix.

articleMarco Paina, Mark Niklas M\"uller, Niels M\"undler-Sasahara, Martin Vechev

Small agents should know what their tools do

EffectWorld defines tools with declared effects and idempotence, enabling label-free trace verification and program-style multi-call turns.

Why it mattersDeclaring each tool's effect, idempotence and read/write resources lets a verifier check agent traces without labels or judges.

articleAlexander Kovalenko, Artem Redchych

UnityBench-Mini3D: Evaluating LLM Agents for 3D Game Development in Unity

UnityBench-Mini3D evaluates coding agents on 77 project-level Unity tasks across seven skill categories, scored by 1,178 human-authored in-engine tests covering scene structure, component configuration and runtime behavior.

Why it mattersGame development tests coding agents on engine APIs, scene state and runtime behavior. This benchmark scores agents with deterministic in-engine tests rather than LLM judges, so failures can be localized.

articleYijun Zhou, Minghao Shao, Ke Zhang, Renkai Yuan, Hongbo Yuan, Bin Hu, Bozhao Qian, Hongxia Shen, Yanhong Liu, Yuting Chen, Yuntao Shi, Zhongxuan Xu, Shuo Mao, Haijia Zhou, Zhendong Li, Tao Zhang, Chenggang Li, Yuxuan Wang, Zinan Zheng, Calvin Zhang, Jia Li, Heng Chang, Jie Feng

Counting the Bandages: On measuring the time that AI saves

Argues that AI time savings should be measured alongside escaped defects.

Why it mattersClock time alone overstates AI savings if defects escape. The paper reanalyzes METR's developer study to show the slowdown estimate shifts from 19% to 34% depending on the estimand, and offers a break-even defect cost for decisions.

articleSolomon Shalom Lijo

Do Coding Agents Still Need a Debugger? Frontier-Model-Based Fault Localization at the Accuracy Ceiling

Researchers seeded 610 single-line faults into 144 programs and measured fault localization by frontier-model agents with no, optional or encouraged debugger use.

Why it mattersIf frontier agents already localize nearly all single-line faults from source alone, adding debugger tooling may cost more than it returns on this class of bugs. It helps decide where to spend tool-integration effort.

articleChristoph Daxerer, Johann Glock, Christian Macho, Rudolf Ramler, Martin Pinzger, Josef Pichler
Artificial Analysis@ArtificialAnlys

Artificial Analysis releases Intelligence Index v5 with FrontierCode 2.0 and Terminal-Bench-Science, upgraded HLE, Omniscience and CritPt, and 60% private evaluation. Claude Opus 5.5 leads at 58, the open-weights gap grows to 12 points, and cost per task varies over 13x.

Why it mattersThe index now relies mostly on private test sets and adds maintainer-merge coding and terminal science tasks, so rankings are harder to game. The open-weights gap widened to 12 points and cost per task spans over 13x.

pydantic-ai v2.63.0 — v2.63.0 (2026-10-27)

pydantic-ai 2.63.0 changes several defaults.

Why it mattersSeveral defaults change in this release, including CodeMode's dynamic tool catalog and the harness reminder interval, which can alter prompt caching and agent behavior after upgrade. Check the compatibility notes before bumping.

repodsfaccini

Particle-1 is now available on AI Gateway

Particle-1, post-trained on DeepSeek V4.1 Flash for software engineering with shorter reasoning, is now on AI Gateway with a 1M-token context.

Why it mattersA new option for coding agents that claims fewer reasoning tokens and fewer repair loops, callable as particle/particle-1 through AI Gateway. The efficiency gains are vendor-described, not independently benchmarked.

articleZachary Chen

Run Paseo agents on Vercel Sandbox

Paseo now has a Vercel Sandbox plugin that runs Codex, Claude Code, OpenCode and Pi in the cloud.

Why it mattersYou can start a coding agent in a Vercel Sandbox from Paseo, resume it from your phone with your laptop closed, and keep the gateway key outside the sandbox.

articleElisabeth Rülke

Claude 529 Overloaded

Explains Claude's 529 overloaded_error versus a 429 rate limit.

Why it mattersA 529 means Anthropic is out of capacity, not that your key or quota is at fault. Retry with backoff, then fail over to another provider or fallback model to keep agents running.

articleOpenRouter editorial sitemap

Github Copilot Fusion Orchestrates Anthropic And Openai Models To Balance Performance And Cost

Artificial Analysis benchmarks GitHub Copilot Fusion, which orchestrates Anthropic and OpenAI models.

Why it mattersCopilot Fusion routes between Anthropic and OpenAI models and reaches near-frontier coding scores at $5.59 per task versus $11.42 for Claude Code with Opus 5.5. That makes routing a credible cost lever for coding agents.

articleArtificial Analysis articles

Bell Personal Assistant Stack

Browserbase's BELL reference architecture traces a coffee order from an iMessage through planning, browser automation, credentials, payments and identity.

Why it mattersShows a concrete layered stack for an assistant that takes a text request through planning, browsing, authentication and checkout. The open sample repo lets you see how messaging, browser, payments and identity integrate.

articleBrowserbase

fx is now available in T3 Code

Vercel's open-source coding agent harness fx is now a built-in provider in T3 Code, the open-source control plane for coding agents.

Why it mattersIf you run coding agents through T3 Code, you can now add fx as a provider from Settings and use it on web, desktop and mobile, with Supervised or Auto approval modes.

articleAndrew Gadzik

Gemini 4 Carbon New Number 1

Artificial Analysis ranks Google's Gemini 4 Carbon first on Intelligence Index v5 and second on its Coding Agent Index.

Why it mattersGemini 4 Carbon leads the Intelligence Index v5 and sits 3 points behind Claude Code with Opus 5.5 on coding agents, at about 23% lower cost per task. Access is limited to selected users for now.

articleArtificial Analysis articles

How We Rebuilt Provider Routing

OpenRouter replaces price-dominant default provider routing with a multi-factor score.

Why it mattersOpenRouter's default routing now scores providers on effective price, throughput, reliability and quality rather than mostly price. Expect faster, more reliable responses at about 6% higher cost unless you pin or sort providers.

articleOpenRouter editorial sitemap

Virtual models now available on AI Gateway

Vercel AI Gateway adds virtual models.

Why it mattersYou can swap models, reasoning effort, fallbacks and ZDR settings centrally without redeploying apps or agents, including coding agents that do not expose those options.

articleRohan Taneja

We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI

Arize's Fuad Ali walks through rebuilds of the Alex agent with custom harness code.

Why it mattersGives tested design decisions for a production agent: put the plan in the harness, gate completion explicitly, and use one small per-user memory file instead of vector stores or knowledge graphs.

videoAI Engineer

Agent Custom Tools Context Skills

Figma opens its canvas-native design agent to more users in beta, adding custom tool creation, broader context about a team's design system.

Why it mattersFigma's design agent is in open beta with custom tools, richer context, and skills, so designers and builders can extend the agent to generate team-specific tools and effects directly on the canvas.

articleFigma Blog

crewAI 1.15.31

crewAI 1.15.31 adds context compaction for conversational flows and fixes GPT-6 stop-parameter handling.

Why it matterscrewAI users should upgrade for fixes to path traversal in output_file template interpolation and other injection issues. The release also stops sending 'stop' to GPT-6 and newer models, and adds context compaction for conversational flows.

repolorenzejay

Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

NVIDIA details how CUDA 13.4 and the open-source GPU Fractioning Operator enforce per-workload GPU memory limits in Kubernetes.

Why it mattersCUDA 13.4 can now enforce per-workload GPU memory limits through MPS. With the GPU Fractioning Operator and KAI Scheduler, several inference workloads can share one GPU on Kubernetes without one exhausting its memory.

articleElizabeth Goodman

Anthropic reports early results from Claude in drug development

Anthropic reports early results from drug development work with six partners and its own programs.

Why it mattersConcrete evidence of Claude running multi-stage scientific work with partners, including autonomous design generation and document review, with measured outcomes.

articleAnthropic

claude-code v2.1.312

Claude Code v2.1.312 adds an oidc.user_id_claim gateway option and click-to-send for queued messages.

Why it mattersClaude Code stops overwriting large CLAUDE.md or rules files it only partly read, and finds path-scoped skills after a Bash cd. Enterprise gateways can now identify users by a claim other than sub, such as Entra ID's oid.

repoashwin-ant

claude-agent-sdk-typescript v0.3.312

Claude Agent SDK for TypeScript 0.3.312 adds autoMode settings types, an unanswered field for AskUserQuestion answers gated on a capability flag.

Why it mattersIf you build on the Claude Agent SDK, you can now type auto-mode permission settings and report skipped or dismissed AskUserQuestion prompts. resolveSettings() also now matches CLI behavior for managed settings.

repoashwin-ant

codex rust-v0.165.0 — 0.165.0

Codex 0.165.0 adds a built-in Vertex AI provider, GPT-6.1 Terra in the model catalogs, session picker repo filters, and mid-conversation reasoning effort changes.

Why it mattersCodex can now run against Google Cloud Vertex AI with model_provider set to gcp. It also lets you change reasoning effort mid-conversation and enables memory v2 for new profiles.

repogithub-actions[bot]

ai ai@7.0.155

AI SDK 7.0.155 drops runtime ModelMessage validation in streamText, generateText, generateObject and streamObject, removing a per-step cost that scaled with conversation length.

Why it mattersIf you use the AI SDK, long agent loops no longer pay a per-step validation cost that grew with message history. Invalid messages are now caught only by TypeScript, not at runtime.

repogithub-actions[bot]

Agents Judgment Calls

Anthropic's randomized trial had 50 causal-inference analysts work with Claude Code while the agent secretly leaned toward randomly chosen options.

Why it mattersExperts handed a coin-flip-assigned lean by their agent adopted it 86% of the time and often reused its arguments as their own. Review processes cannot assume a human is independently making the judgment calls.

articleAnthropic research

Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI

Arize's Mikyo King argues for giving agents a code sandbox instead of tool catalogs, shows how Phoenix implements it, covers the operational costs.

Why it mattersReplacing a large tool catalog with a sandbox where the agent composes tools in code cut tokens by 43% and cost by 31% in Arize's Harbor evals, with the tradeoffs around secrets and permissions spelled out.

videoAI Engineer

An index of the vibe-coding frontier. Corrections welcome.