Vibeleaderboard
Index — Latest Intelligence

Intel

Page 22

Template-Free Dynamic Invariant Detection

A dynamic invariant detector that drops fixed templates and lets an LLM propose pre- and postconditions.

Why it mattersShows how to let an LLM propose program invariants safely, with guards against side effects and infinite loops in generated checks, useful for anyone running LLM-written code at runtime.

articleErfan Arvan, Michael D. Ernst, Martin Kellogg

Seq@K: Evaluating Language Models Under Sequential Feedback

Seq@K scores whether a model solves a task within K attempts given feedback after each failure.

Why it mattersPass@1 misses how well a model uses feedback after failures, which is how agents work. Seq@K shows model rankings can change under sequential feedback, so benchmark choice may alter which model you pick.

articleThomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong
Kimi_Moonshot@Kimi_Moonshot

Moonshot's Kimi added 3D Make, which connects the assistant to Blender and CAD tools to create models, assemble scenes and develop games. It is available on the Kimi site and Kimi Work. The announcement gives no benchmarks or technical detail.

Why it mattersKimi's agent can now drive Blender and CAD tools directly to create models, assemble scenes and build games. Engineers working on tool-using agents get another example of agents operating professional 3D software.

Artificial Analysis@ArtificialAnlys

Artificial Analysis adds HLE-Diamond, the cleaned 1,000-question Humanity's Last Exam, to its index. Gemini 4 Carbon leads at 69%, models do better on reasoning than knowledge, open-weight models trail by over 30 points, and top effort settings multiply cost per task.

Why it mattersShows what extra effort buys: Opus 5.5 gains 26 points from low to max effort at about 35x the cost per task. It also shows models are much stronger on reasoning than knowledge questions.

Vercel MCP now supports MCP events

Vercel MCP now publishes platform events. An agent subscribes once by team, project and environment, and receives signed Standard Webhooks deliveries.

Why it mattersAgents can react to deployment.succeeded or deployment.error events through a managed, self-expiring subscription instead of polling, without consuming the 40 account webhook slots.

articleJosh Souphanthong

Gemini 4 models land on Cognition's FrontierCode 2.0

Cognition adds Gemini 4 Carbon (40%, $4.04 per task) and Gemini 4 Argon (26.3%, $6.41) to FrontierCode 2.0, a 132-task mergeability benchmark written by open-source maintainers.

Why it mattersFrontierCode 2.0 grades whether a maintainer would merge the PR, not just whether tests pass.

articleCognition

How Coinbase Used Code Connect To Shrink Token Costs

Coinbase's design system team evaluated Figma Code Connect with coding agents through the Figma MCP server and its own agent skills.

Why it mattersGiving a coding agent a mapping to your real production components cut tokens and time by roughly a fifth in Coinbase's evaluation while improving code quality. This is evidence for investing in agent-facing design-system context.

articleFigma Blog

The Figma Agent Is Here

Figma launched a design agent built into the canvas, tuned for editing Figma files with knowledge of your components and tokens.

Why it mattersFigma now ships a native agent on the canvas with context on components and tokens. The post clarifies when to use it versus the MCP server and use_figma for moving code and designs between canvas and repo.

articleFigma Blog

The Figma Canvas Is Now Open To Agents

Figma opened its canvas to AI agents.

Why it mattersAgents can now write to Figma files via the MCP use_figma tool, linked to your design system, and skills carry team decisions as context. It is free in beta and will become usage-based paid, which affects cost planning.

articleFigma Blog

claude-code v2.1.314

Claude Code v2.1.314 adds a permission that blocks file writes outside working directories in all modes, IdP-group routing for gateway upstreams, Bedrock guardrail tracing, idle compaction in VS Code and Desktop.

Why it mattersClaude Code can now refuse Write, Edit and NotebookEdit outside working directories in every permission mode, and gateway upstreams can be restricted to IdP groups. Several Bedrock and Foundry failure modes are also fixed.

repoashwin-ant

claude-agent-sdk-typescript v0.3.314

Claude Agent SDK for TypeScript v0.3.314 adds sandbox_required_unavailable as a startup failure reason, a permissions setting to block writes outside working directories, an alpha prewarm() entry point.

Why it mattersThe SDK now exposes a startup failure reason for unavailable required sandboxes, a setting that blocks writes outside working directories, and an alpha prewarm(). It also fixes settings leaking in from files excluded via settingSources.

repoashwin-ant

How we use Insights to improve the agent behind Insights

Datadog engineers show how trace analysis of their own agent exposed cost drivers, such as search tools returning more content than needed that is then resent on every turn.

Why it mattersOversized tool results get resent in every later model request, inflating token cost. Trace-level analysis surfaces these patterns and tool-call failures so you can fix them with evidence instead of guesswork.

articlewww.datadoghq.com

Claude Opus Vs Sonnet Vs Haiku

OpenRouter ran Opus 5.5, Sonnet 5.5 and Haiku 5.5 through identical coding tasks and a Claude Code bug fix. All passed, so cost separated them.

Why it mattersOn the tested tasks, Haiku 5.5 cost 19 to 30 times less than Opus 5.5 with the same pass rate, so default to a cheaper tier and reserve Opus for long, ambiguous work. Verify on your own tasks.

articleOpenRouter editorial sitemap

Claude Code Cost

OpenRouter uses 30 days of real Claude Code sessions to give median per-session API costs by model and length.

Why it mattersA short Claude Code session costs about $0.31 median on Opus 5.5 and $0.16 on Sonnet 5.5, mostly from cached context, so keeping sessions short and routing sub-agents to Haiku cuts spend.

articleOpenRouter editorial sitemap

Ling 3.1 Flash 1019 now available on AI Gateway for free

Vercel adds InclusionAI's Ling 3.1 Flash 1019 to AI Gateway, free until November 3, 2026.

Why it mattersLets you try a 1M-context, tool-use-oriented model through one gateway ID with no charge until November 3, 2026, after which published rates apply. Plan evaluation runs before then.

articleJerilyn Zheng

Together AI adds US-only serverless inference endpoints

Together AI adds a US-only serverless endpoint for seven open models, including Kimi K3, GLM-5.3 and DeepSeek V4.1 variants. Same key and model IDs, 1.25x token pricing.

Why it mattersLets you keep inference for supported open models inside the US by changing the base URL, at a 25% premium. It only guarantees processing location, and requests fail rather than fall back, so retry handling must stay on the same endpoint.

articleTogether AI

Offline Evaluations Sre Agent

Mercor describes converting closed production incidents, with root-cause analyses, into offline eval tasks for SRE agents.

Why it mattersShows how to turn every closed incident into a graded eval task with a frozen world and human-confirmed answer key, which let one team compare two agents and cut cost per alert more than 3x.

articleMercor research blog

API credits benefit for eligible ChatGPT subscriptions

OpenAI now gives $100, $200 and $500 ChatGPT subscribers equivalent credits to apply to an API organization, matching a benefit Anthropic launched earlier this month for Max and Team plans.

Why it mattersChatGPT subscribers on $100, $200 or $500 plans now get matching API credits for one API organization, mirroring Anthropic's Max and Team benefit. Existing subscriptions can cover prototype and demo API spend.

ClaudeDevs@ClaudeDevs

Anthropic raises the Claude API request size cap for Messages and Token Counting from 32 MB to 128 MB, and lifts the per-image limit from 2000 to 3000 px for requests with over 20 images. Cloud platform deployments keep 20 to 32 MB limits.

Why it mattersLong computer-use sessions and large PDFs no longer hit the 32 MB request cap before the context window fills. The 128 MB limit applies only to the first-party API, so cloud-platform users still face 20 to 32 MB.

Artificial Analysis@ArtificialAnlys

Artificial Analysis reports AA-PersonalityBench results for Grok 4.5, 4.6 and 4.7. Slop dropped from the 68th to the 6th percentile, sycophancy from 18 to 1, and sincerity rose from 26 to 90. Methodology was linked but not reviewed.

Why it mattersAA-PersonalityBench shows Grok's slop percentile falling from 68 to 6 and sycophancy from 18 to 1 across 4.5 to 4.7. Engineers picking a model for user-facing agents get a measured signal on flattery and filler.

We ran AI agent swarms in Minecraft and Mindustry

YC Labs ran swarms of Claude and GPT agents in Minecraft and Mindustry, including 50 agents building a colony over 3 hours.

Why it mattersFifty agents given one shared goal formed a de facto leader, ration ledgers and written rules within hours. The open-source harness lets you rerun and study swarm coordination yourself.

articlesandslash
Epoch AI@EpochAIResearch

Epoch AI reports GLM 5.3 autonomously built working Firefox exploits from unshipped public fixes for six versions, about $1,000 each, fastest in 12 hours. It also found an unpublished bug that Mozilla patched. Chrome attacks were not reproduced. Methodology was not read in full.

Why it mattersOpen-weight models can now weaponize published but unshipped fixes cheaply, so the window between a public patch and a release is a real exposure. Shortening it becomes a defensive priority for software maintainers.

OpenAI@OpenAI

OpenAI outlines its expanded monitoring of frontier agents, combining activation probes, reviewer models and investigative agents across RL runs and internal Codex use. A replay of the Hugging Face incident would have alerted about 30 hours before the breach.

Why it mattersOpenAI describes a three-layer monitor (probes, reviewer models, investigative agents) now covering all frontier RL runs and internal Codex use.

codex rust-v0.166.0 — 0.166.0

Codex 0.166.0 adds GPT-6.1 Terra to bundled, Vertex AI and Bedrock catalogs.

Why it mattersCodex 0.166.0 adds GPT-6.1 Terra to its model catalogs (including Vertex AI and Bedrock) and fixes reasoning-token accounting on resume and fork, which previously risked premature compaction.

repogithub-actions[bot]

GenesisDB – The event sourcing database, is now open source

GenesisDB 2.0 drops its Community/Enterprise split and open-sources the server, SDKs and VS Code extension under MIT, adding a built-in MCP endpoint for AI agents alongside its existing hash-chained audit trail and GDPR erasure.

Why it mattersAn event-sourcing database with hash-chained audit trails and GDPR erasure is now fully free and MIT-licensed with no feature gating, and it exposes an MCP endpoint so agents can query it directly.

articlepatriceckhart

Intelligence Index v5 moves benchmarks toward private, agentic tests

Artificial Analysis released Intelligence Index v5, adding FrontierCode 2.0 and Terminal-Bench-Science, upgrading three evaluations.

Why it mattersIndex v5 shifts weighting toward agentic coding and science, and 60% of it now uses private test sets, reducing contamination. Scores are not comparable with v4, so model comparisons must be re-based on the new weights.

articleArtificial Analysis

Why Humans Can't Keep Up With AI-Written Code - Tariq Shaukat, Sonar

Sonar's CEO argues agent-generated code needs factory-style quality control.

videoAI Engineer

opencode v2.2.2

opencode v2.2.2 enables experimental image generation by default when a provider offers image models, adds a direct worktree picker and a unified quote and annotation editor.

Why it mattersopencode 2.2.2 lets agents call an image-generation tool automatically when a connected provider supports image models, with the preferred model configurable.

repogithub-actions[bot]

Anthropic's self-assessment: cheap misuse monitoring using the model itself

Anthropic details self-assessment: after each reply Claude emits a short structured safety judgment from the cached conversation.

Why it mattersSelf-assessment reuses the cached conversation to flag misuse for about 10 extra tokens, adding under 2% compute. Teams building agent monitoring get a cheap monitor design, plus evidence that it works offline but is weak as a blocker.

articleAnthropic

Managed Agents

Anthropic explains how Managed Agents separates the session log, harness and sandbox into swappable interfaces, borrowing the OS abstraction pattern.

Why it mattersHarness logic that compensates for model weaknesses goes stale as models improve. Separating session log, harness loop and sandbox behind stable interfaces lets you swap each one without rewriting the others.

articleAnthropic research
Claude@claudeai

Anthropic details a round of Claude mobile app updates: artifacts open beside chat on iPad, multi-account switching, a resizable laptop-style Android layout, 24 added languages, faster voice mode, CarPlay support, and pull-to-search coming soon.

Why it mattersClaude mobile now supports multiple signed-in accounts, voice mode in CarPlay, 33 languages, and desktop-style Android on Chromebooks and foldables. Relevant if you use Claude across work and personal accounts or on non-iOS devices.

ai @ai-sdk/xai@5.0.33

The AI SDK's xAI provider release details a fix making the top-level reasoning option resolve the same way across providers.

Why it mattersReasoning-effort values like 'xhigh' or 'none' now map predictably to each provider's closest supported level instead of being silently dropped or rounded inconsistently.

repogithub-actions[bot]

ai ai@7.0.157

Vercel's AI SDK core package graduates LanguageModelStreamPart and filterActiveTools out of experimental status and fixes spurious Next.js Edge Runtime warnings from telemetry's Node.js runtime detection.

Why it mattersTwo previously experimental AI SDK APIs are now stable, so code depending on them no longer risks breaking changes, and an Edge Runtime warning from telemetry tracing is fixed.

repogithub-actions[bot]
OpenRouter@OpenRouter

OpenRouter shipped a rebuilt provider router that ranks providers by quality relative to the actual cost of a request, not mostly by price. Better-performing providers win more traffic. The linked methodology post was not collected, so scoring details are unknown.

Why it mattersOpenRouter now routes by quality of inference relative to real request cost rather than mostly price, so traffic shifts toward faster, more reliable providers. Engineers using it may see different latency and reliability.

Hy-Nomas: open-weight 8B interactive world model with open benchmark

Tencent's Hy-Nomas is an 8B video world model taking free-form text and camera control across turns.

Why it mattersShows how to keep a video model's actions scoped to their time window across turns, with open weights, a LoRA adaptation recipe and a reproducible multi-turn benchmark. Useful if you build on or evaluate world models.

Artificial Analysis@ArtificialAnlys

Artificial Analysis introduces its Progress Index, an item response theory scale covering models since 2022. Findings: frontier gains of about 65 to 70 points a year, open weights lagging proprietary models by a median of about 4 months, and new frontier highs roughly every four weeks in 2026.

Why it mattersA single capability scale across years shows open-weights models trail the closed frontier by a median of about 4 months, and that prices at a given capability level have fallen by orders of magnitude within months.

Artificial Analysis@ArtificialAnlys

Artificial Analysis benchmarks Phonic 1 (High) as #1 on its Speech to Speech Index at 87.1, narrowly ahead of Grok Voice 3.0. It leads Tau Voice agentic scoring, is #8 on arena preference, and costs $4.20 per hour of input audio.

Why it mattersPhonic 1 leads Tau Voice (79.9%, 5.4 points ahead) for policy-following and tool use in long voice tasks, at 0.61s time to first audio and $0.07 per minute. Useful when choosing a speech-to-speech model for voice agents.

Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale

Meta reports moving Catching JiTTests, LLM-generated tests produced at diff time to catch bugs before code lands, from research prototype into production infrastructure.

Why it mattersShows LLM-generated tests run per code change in a very large codebase, with fewer false positives and less load than traditional testing. Gives teams a reference design for gating changes with generated tests.

article

Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning

Latent Space covers Elorian's DecoyCount benchmark, which tests whether models truly count objects or use visual shortcuts.

Why it mattersFrontier vision models score below 50% on a counting benchmark built with decoys, showing that stronger general reasoning does not guarantee reliable visual counting. Check this before trusting a model for perception tasks.

articleRichard MacManus

Science Behind the Open Cyber Frontier with Mistral Large 4

Hugging Face talks with Mistral researchers about Mistral Large 4.

videoHugging Face

LangChain Managed Deep Agents gain browser use with human handoff

LangChain added browser use to Managed Deep Agents.

Why it mattersManaged deep agents can now drive a browser on sites with no API, with thread or user scoped sessions and a human handoff for login, CAPTCHA and 2FA. Scope choice determines whether cookies and logins are shared across conversations.

articleLangChain
Artificial Analysis@ArtificialAnlys

Mistral releases Large 4 weights (1.05T total, 52B active, 1M context). Artificial Analysis scores it 26 on its index, with 39% fewer output tokens than the Preview, 28% hallucination rate, halved pricing and a license that restricts hosted-service use.

Why it mattersMistral Large 4 is the strongest non-Chinese open-weight model on this index, at roughly half the tokens per coding-agent task of leaders. The MEOW license requires a separate deal if you run a model-as-a-service business.

Mistral AI@MistralAI

Mistral released weights for a new Le Chonk model targeting agentic coding and cyberdefense. It claims the top open-model spot on FrontierCode and a Pareto-efficiency lead on Artificial Analysis. Weights are on Hugging Face. Benchmark details come from the announcement only.

Why it mattersMistral released new open weights for Le Chonk, pitched at agentic coding and cyberdefense, downloadable from Hugging Face. Teams wanting a self-hostable coding model get a new option, though the benchmark claims are Mistral's own.

transformers v5.22.0 — v5.22.0: Mistral Large 4 (Le Chonk)

Transformers 5.22.0 adds Mistral Large 4.

Why it mattersTransformers v5.22.0 adds Mistral Large 4, a 1T-parameter open MoE with 49B active parameters, 1M context and per-request reasoning effort, so it can be loaded and tested in existing pipelines.

repoLysandreJik

Mistral Large 4 (Le Chonk) is now Open Weight

Mistral releases the weights of Mistral Large 4, a 1T-parameter multimodal model with three reasoning modes.

Why it mattersA 1T-parameter open-weight model claiming top open coding scores can be self-hosted and fine-tuned, giving agent builders a frontier-class alternative to closed APIs. Benchmark claims are Mistral's own.

articlekuberwastaken

ZwZ2 trains vision models to connect evidence across image regions

Ant Group and Shanghai Jiao Tong researchers synthesize multi-region visual questions, filter them by cross-model agreement and region ablation, and RL-train Qwen VLMs.

Why it mattersEvidenceConnect shows that verified single-image, multi-region questions used for RL training can improve spatial reasoning, long-context and GUI-agent benchmarks without multi-image data.

Software engineering at Google & CS education with Titus Winters

Gergely Orosz interviews Titus Winters, who helped shape today's software-engineering curriculum and Google's engineering practices, on why he now says to burn parts of it down and rethink what the discipline teaches.

Why it mattersA veteran who helped write the modern software-engineering curriculum argues much of it is outdated, with concrete critique of how CS education maps (or fails to map) to real engineering practice.

articleGergely Orosz

Granite Switch 4.1.1 — Update models like you update software

IBM's Granite Switch 4.1.1 is a backwards-compatible patch.

Why it mattersIBM shows modular adapters on a single base checkpoint being updated like software patches, with no base-model recompute or migration. This points to cheaper ways to version and ship model capabilities.

articlehuggingface.co

Deepgram acquires Daily and Pipecat, introduces Real-Time Context Standard

Deepgram is acquiring Daily, maker of the Pipecat voice agent framework, which stays BSD open source under a planned independent foundation.

Why it mattersPipecat, a widely used voice agent framework, now sits under Deepgram while staying BSD licensed with a planned independent foundation. The Real-Time Context Standard adds metadata so tone and speaker cues survive between models.

articleDeepgram

Internal Ops Agents

ElevenLabs describes its Slack-based internal agents for sales, support triage and data analysis.

Why it mattersIt shows a working pattern for internal agents that act on requests, not just answer them: dedupe, enrich, prioritize and route tickets using an uploaded ownership document.

An index of the vibe-coding frontier. Corrections welcome.