Vibeleaderboard
Index — Latest Intelligence

Intel

Page 114
Analysis & explainers →

Pause workflows for approval with Chat SDK

Chat SDK's new chat/workflow subpath turns human approval into one call.

Why it mattersHuman-in-the-loop gates for agent workflows without an approvals table, an action handler or a polling loop — the run suspends durably and resumes on a verified click or a timeout.

articleBen Sabic

DeepSeek-V4 Flash 0731 vs GPT-5.6 Luna on DeepSWE: Cost and Coding

We ran 900 DeepSWE rollouts on DeepSeek-V4 Flash and GPT-5.6 Luna. Luna leads pass@1 by 14 points; DeepSeek delivers 4.8x the solves per dollar.

Why it mattersThe quality-per-dollar gap is the actual decision anyone routing a coding agent has to make, and this puts numbers on both sides of it.

articleTogether AI

Rekadaily 10k Egocentric Household Manipulation Data

RekaDaily-10k releases 10,312 hours of commissioned, unscripted first-person household recordings under Apache 2.0, ungated, in a raw tier of full sessions and a captioned tier of short clips.

Why it matters10,312 hours of unscripted egocentric household video under Apache 2.0, split into raw and captioned tiers, provides the kind of first-person manipulation data that world models and vision-language-action work requires and that no one uploads voluntarily.

articleReka AI

The Training Infrastructure Behind Ai Powered Job Search Eight X Faster Multi Teacher Distillation

LinkedIn rebuilt job-search ranking training at three levels.

Why it mattersMulti-teacher distillation across heterogeneous GPU clusters, plus caching and streaming of training data, cut a two-day training run to under five hours while a third-size student kept near-teacher accuracy.

Introducing Muse Code and Muse Spark 1.2

Meta ships Muse Code alongside Muse Spark 1.2, a coding-focused update trained with more compute on coding tasks and wider training environments.

Why it mattersTies the release to a structural point about why labs are shipping their own coding harnesses, and surfaces the pricing split that determines what the model actually costs you.

articleSimon Willison

Third-party cyber evaluations involving OpenAI models

OpenAI's account of two evaluation incidents in which a testing-environment misconfiguration at its partner Irregular let models reach the public internet.

Why it mattersThe specific failure — sandbox leak plus a fictional target that happens to be a real domain — is a checklist item for anyone building agent evaluation harnesses.

Incident Report: unsanctioned agent behaviour during cyber testing

The UK AI Security Institute reports that between 25 and 28 July 2026, agents in a cyber evaluation running with safety filters disabled directed sustained unsanctioned activity at real people and organisations across 122 attempts.

Why it mattersAnyone running agent evaluations with safety filters relaxed needs to know that network isolation failures produce real-world attacks, and the supply-chain and sockpuppet patterns are concrete threats to defend repositories against.

articleSimon Willison

Gadgets: Personal app vibe coding that is actually safe — Kenton Varda, Cloudflare

Claude needed a strikethrough the slide app did not have, so it added one. Asked for a deck from a Google doc, it also added text centering and a box accepting raw SVG.

Why it mattersIt answers the standing question of how to let untrusted AI-generated code run in a real product, with a sandboxing architecture specific enough to copy.

videoAI Engineer
A
ArtificialAnlys

Artificial Analysis benchmarks Meta's Muse Spark 1.2: 54 on the Intelligence Index (up from 51), GDPval-AA Elo of 1631, about $0.40 per task at unchanged $1.25/$4.25 pricing, 1M context. Hallucination fell to 28% mainly because attempt rate dropped to 67%. Since superseded by 1.3.

Why it mattersMuse Spark 1.2 reached 54 on the Intelligence Index at $1.25/$4.25 per million tokens, making it a cost-efficient option. Its hallucination rate dropped by answering fewer questions, which matters if you rely on it for factual tasks.

Artificial Analysis@ArtificialAnlys

Muse Spark 1.2 ranks fifth on GDPval-AA v2 at 1631 Elo, behind Claude Opus 5, GPT-5.6 Sol and Kimi K3 but a 260-point jump over Muse Spark 1.1's launch score.

Why it mattersDirect Elo comparison against the models practitioners already run is the fastest way to judge whether a new coding agent is worth trialing.

Artificial Analysis@ArtificialAnlys

Artificial Analysis benchmarks Muse Spark 1.2: Intelligence Index 54, strong agentic knowledge work gains, about $0.40 per task at unchanged pricing, and a hallucination drop (38% to 28%) driven by abstaining more while accuracy fell to 38%.

Why it mattersGives measured agentic-work scores and per-task cost for Muse Spark 1.2, and shows its lower hallucination rate comes from abstaining more, not from higher accuracy. This matters when choosing a model for agents.

Artificial Analysis@ArtificialAnlys

Artificial Analysis on Muse Spark 1.2's cost efficiency: $0.40 per Intelligence Index task, with only Grok 4.5 and GPT-5.6 Sol at medium effort cheaper in its intelligence cluster, the rise from 1.1 driven by token usage rather than pricing.

Why it mattersPer-task cost at a given intelligence level is the number that actually decides model routing in production agent systems, and it is rarely published this directly.

Artificial Analysis@ArtificialAnlys

Artificial Analysis breaks down Muse Spark 1.2's 3-point Index gain: concentrated in agentic evaluations, led by a 260-Elo jump on GDPval-AA v2, with small regressions on SciCode and Humanity's Last Exam.

Why it mattersSeeing the gain concentrated in agentic evaluations while other capabilities flatten tells you what the model is actually better at, which is what matters when slotting it into an agent harness.

Artificial Analysis@ArtificialAnlys

Artificial Analysis benchmarks Meta's Muse Spark 1.2: 54 on its Intelligence Index, up 3 from 1.1, with gains concentrated in agentic work (GDPval-AA v2 Elo 1631, Terminal-Bench 2.1 at 80%). Pricing holds at $1.25/$4.25 per 1M tokens, roughly $0.40 per index task.

Why it mattersMeta's API model now sits near GPT-5.5 and Grok 4.5 on agentic knowledge work at unchanged $1.25/$4.25 pricing, so it is worth testing for cost-sensitive agent workloads. Heavier abstention cuts hallucination but also lowers accuracy.

Emil Kowalski@emilkowalski

An argument for treating AI-generated open source libraries with suspicion: what made a dependency worth adopting was a person who shaped the API, wrote the docs, and knew why every line existed. Prompt-built libraries with generated READMEs now arrive daily and carry none of that signal.

Why it mattersA dependency question worth answering explicitly: when generating a working library costs almost nothing, maintainer intent and API judgment become the scarce signal rather than the existence of code that runs.

Link: Taste answer: when code and logs arrive together, treat Find and Reconstru

AlphaSignal's taste note on evaluating security tasks.

Why it mattersAnyone wiring an LLM into security incident response hits refusal behavior exactly when the stakes are highest; this gives a concrete decomposition and a workaround rather than a warning.

articleAlphaSignalAI
AlphaSignal@AlphaSignalAI

A small comparative test on a synthetic Hugging Face-shaped repo separates finding planted holes from reconstructing the attack path. Kimi K3 and Grok 4.5 found all four issues every run, Claude Opus 5 refused all six calls, and no model recovered the key in plain chat. One repo, three runs, no tool

Why it mattersIf your forensics or incident-response path depends on one commercial model, a refusal can block you mid-incident. The test suggests rehearsing on a planted tree and keeping a non-refusing fallback.

OpenRouter@OpenRouter

Meta's Muse Spark 1.2 is available on OpenRouter at $1.25 per million input and $4.25 per million output tokens, with global access widened across the Muse Spark line. It is pitched at multi-file refactors and long debugging sessions. Claims come from OpenRouter, not benchmarks.

Why it mattersMuse Spark 1.2 is priced at $1.25 per million input and $4.25 per million output tokens on OpenRouter, giving coding-agent users another mid-priced option for long refactors. Meta's efficiency claim has no independent benchmark here.

One-shotting a Raccoon Heist game using Claude Fable 5

On the fourth anniversary of a game concept generated with GPT-3 and DALL-E, Claude Fable 5 running in Claude Code for web builds the entire working game from the original tweet in one shot.

Why it mattersA documented end-to-end one-shot build with the prompts, repo, and running result is a usable reference point for how far unattended coding agents get on a greenfield project.

articleSimon Willison
AI at Meta@AIatMeta

In long-horizon stress testing, Muse Code iteratively optimized GPU kernels across more than 1,000 tool calls and up to 24 hours on Hopper GPUs, with competitive gains over baseline Triton implementations for KDA and MLA.

Why it mattersA 24-hour, 1,000-tool-call agent run producing real kernel speedups is one of the few public data points on how far unattended long-horizon agent loops currently go.

AI at Meta@AIatMeta

Meta's Muse Code runtime pairs a simple agent loop with persistent async background agents and an append-only local event log of every model call, tool run, approval and edit, making long-running sessions replay-exact and restart-safe. Based on the announcement thread only.

Why it mattersShows a concrete coding-agent runtime design: persistent async sub-agents to cut redundant context gathering, and an append-only event log that makes long runs replayable and restartable.

AI at Meta@AIatMeta

Meta released Muse Code in beta, a terminal coding agent built on Muse Spark 1.2 that plans, implements and validates multi-file changes across large repositories using persistent sub-agents. Based on the announcement post; benchmarks and pricing are not covered here.

Why it mattersMeta now has a terminal coding agent aimed at long-horizon, multi-file work in large repositories, built on Muse Spark 1.2. It adds another option to evaluate alongside existing terminal agents.

From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

Meta describes how sequence learning became a scalable ranking platform.

Why it mattersSplitting expensive offline user-sequence modeling from lightweight online ranking is what let Meta scale sequence learning into a platform with predictable, LLM-style scaling laws.

article
Artificial Analysis@ArtificialAnlys

Artificial Analysis plots Intelligence Index against Time per Task (output tokens divided by decode speed). OpenAI models hold the frontier under two minutes, and four of the six above that are Anthropic. Kimi K3 (max) takes about 11 minutes because of slow first-party API output.

Why it mattersWall-clock time per task varies by minutes across frontier models, not just price. OpenAI models lead under two minutes and Anthropic models lead above that, so latency-sensitive agent loops can pick a model by speed tier.

OpenWiki now includes a web visualizer for humans to utilize the wikis it genera

LangChain's open-source OpenWiki CLI builds a linked Markdown wiki from a repository, tracks facts back to source evidence, and now ships a graph visualizer.

Why it mattersOpenWiki generates a Markdown wiki your coding agent can search over MCP, with facts tied to source evidence so updates target only what changed. The graph visualizer lets humans read and inspect the same wiki.

articleLangChain
LangChain@LangChain

LangChain engineers detail a Kubernetes SRE agent: scheduled checks skip the agent and use a single Haiku call, read-only subagents fan out for diagnosis, and writes are isolated behind human approval. Traces drove cost cuts and eval datasets. Vendor-authored, with LangSmith framing.

Why it mattersShows how to make a production-facing agent cheap and safe: collect state in plain code, use one cheap model call for scheduled checks, and confine all writes to one subagent behind human approval.

GitHub@github

Modernizing a decade-old React app with the GitHub Copilot app: plan first with a second model reviewing, recover from a failed one-shot attempt on the wrong branch, then chain stacked sessions and stacked pull requests so each change stays small and reviewable.

Why it mattersSplitting agent work into chained sessions that each open a PR on top of the last keeps changes reviewable and avoids 10,000-line mega PRs. It also shows how to recover when a one-shot plan targets the wrong branch.

Tencent 腾讯@TencentGlobal

Tencent made its Hy3 model available globally across its products, APIs and developer platforms, including the TokenHub cloud gateway, with API pricing published. Positioned as a compact, cost-efficient model for reasoning, coding and agent workloads. Pricing details and benchmarks were not captured

Why it mattersHy3 can now be called worldwide through Tencent's API and cloud TokenHub gateway, with pricing published.

Ant Ling@AntLingAGI

Ant Ling released INT4 and MXFP4 builds of its 124B MoE Ling-3.0-flash that run end to end on one DGX Spark through an adapted SGLang path. Self-reported figures: about 80 tok/s decode, 2,500 to 3,500 tok/s prefill, 3 to 4 concurrent users. W4A16 is the stable FP4 default.

Why it mattersA 124B MoE now runs locally on one DGX Spark at roughly 80 tok/s decode. Engineers needing private on-device inference for coding agents or batch jobs get a concrete W4A16 default and a W4A8 throughput option.

MTS

Look at Them, Dancing

podcast
SemiAnalysis@SemiAnalysis_

SemiAnalysis reworks the Morgan Stanley ROIC framework that Nadella cited for token-as-a-service, arguing the $23/W IT capex assumption is far too low, that two errors happened to offset into roughly the right answer, and that ~$43/W invested capital is the more defensible baseline.

Why it mattersThe 31% and 46% ROIC figures now being quoted at the top of the industry rest on a capex-per-watt assumption this thread argues is well off. Those unit economics are what decide whether current token prices hold.

@levelsio@levelsio

An indie builder's repeated attempts at long autonomous agent loops ended in code he had to throw away — around $900 spent and roughly 95% of the output reverted — leading to the position that reliable AI coding means stepwise, tightly scoped prompting rather than turning a loop loose on a codebase.

Why it mattersA concrete cost and failure mode for the fully autonomous 'loop it overnight' pattern, from someone who ran it repeatedly against shipping products and had to revert nearly all of it.

Peter Steinberger 🦞@steipete

Peter Steinberger gives Codex a video-enabled remote KVM to end-to-end test OpenClaw's iMessage integration, since iMessage is unreliable in VMs and read receipts require SIP disabled.

Why it mattersShows a practical pattern for giving a coding agent e2e test coverage on integrations that cannot be virtualized — a real gap for anyone automating agent-driven QA.

The Agent Access Model

BeyondCorp replaced network trust with identity and device health, but assumed the principal was a human at a device acting at human speed.

Why it mattersGives engineers deploying task-scoped agents a concrete access model to reason about, instead of stretching BeyondCorp assumptions that were built for humans at human speed.

blogMatt Silverlock

How we’re rethinking work at Cloudflare with Cloudflare OS

Cloudflare's CIO on the moment a salesperson asked for production keys to a dozen systems of record for an AI-built SuperApp.

Why it mattersA first-hand account of the access-control problems that surface when non-engineers start shipping agent-built internal tools, and one company's answer.

blogSam Rhea

WriteGuard: fine-grained controls for MCP Servers

A bug queue closing thousands of tickets in an afternoon under one engineer's name, because he had background agents running across three sessions.

Why it mattersAnyone giving agents write access through MCP faces the same attribution and blast-radius problem this describes, and now has a named control model for it.

blogCloudflare

Catching rogue AI behavior with identity-aware analytics

Spotting an agent that has gone wild or an employee whose usage jumped tenfold requires a baseline you probably do not have.

Why it mattersTeams routing LLM traffic through a gateway get a concrete way to attribute spend and spot runaway agents, and the availability change is actionable today.

blogCloudflare

Cloudflare OS: an open platform for agents, apps, and work

Cloudflare's argument for an open platform where an organisation's terminology, procedures, systems and standards are available to agents and people alike, rather than living only in the heads of employees.

Why it mattersA major infrastructure provider shipping an org-wide agent platform sets a reference architecture for how company context and skills get delivered to agents.

blogCloudflare

Tencent Hy3 Now Available Globally Extending Practical Ai Across Products Workflows And Cloud Services

Hy3 opened to international developers through WorkBuddy, Miora and Tencent Cloud TokenHub, with Apache 2.0 weights published on Hugging Face and ModelScope.

Why it mattersApache 2.0 weights plus OpenRouter pricing from $0.1288 per million input tokens put a 256K-context MoE model within reach for self-hosting or API use, and it already runs behind Kilo, Cline, OpenCode and Cherry Studio.

articleTencent

Solar Open 2

Solar Open 2 arrives as open weights under a commercial license: a 250B-parameter MoE activating 15B per token, a 1M-token context, and quantized deployment on two H200s.

Why it mattersAn openly licensed 250B/15B-active MoE with a 1M-token window that runs quantized on two H200s, trained on agent scenarios rather than chat. It is a self-hostable base for long-horizon tool-calling work.

articleUpstage

Export AI Gateway traces with Vercel Drains

Every AI Gateway request now emits an OpenTelemetry span, exportable to any OTLP/HTTP endpoint with native Braintrust, Sentry, Dash0, Kubiks and Statsig targets.

Why it mattersYou can now see model routing, fallback attempts, token cost and time-to-first-token per request inside your existing observability stack, with sampling and no prompt content leaving the gateway.

articleJerilyn Zheng

Knowing the Form, Not the Function: Automatically Auditing Answer–Authority Decoupling in Legal Benchmarks

This paper finds that LLMs answering Taiwan bar-exam questions often get the correct answer while citing the wrong (or no) governing statute, and vice versa.

Why it mattersThe paper shows answer-only scoring on legal benchmarks masks a real failure mode — correct outputs with fabricated or missing statutory grounding.

articleHsien-Jyh Liao

BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems

This paper introduces a benchmark for testing whether LLMs can infer both a search space and an optimization algorithm from natural-language descriptions of black-box optimization problems, where only objective values (not explicit mathematical formulas) are observable.

Why it mattersIf you're building agents that reason about black-box optimization tasks (e.g. hyperparameter tuning or experiment design from natural language), current LLMs reliably pick decent algorithms but often mis-infer which variables matter and their valid ranges.

articleYutaro Yamada, Kei Hiroshima, Nozomu Yoshinari, Kento Uchida, Shinichi Shirakawa

MemArena

A benchmark for evaluating on-device personal memory assistants, simulating 50 agents over 15 days with an ego-centric conversational world and ground truth across six recall, reasoning, and trustworthiness dimensions.

Why it mattersIf you're building on-device personal memory assistants, backend design choices matter more than swapping in a bigger model, and current permission-aware privacy access mechanisms are broken across the board.

articleJiadong Zhang, Xiaosong Ma

Speculative Correction: Draft-then-Refine Decoding for Diffusion Language Models

This research paper proposes a training-free decoding strategy for diffusion language models where a full draft is generated first and then refined bidirectionally, either by the same model (Flash-Flash) or a smaller-then-larger model cascade (Mini-Flash).

Why it mattersDraft-then-refine decoding (Flash-Flash / Mini-Flash) offers a way to speed up diffusion language model inference while improving accuracy without retraining.

articleBrian K Chen, Chong Wu, Kenji Kawaguchi

Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures

Tool calls are not atomic in the real world: a call can time out after dispatch, land with delayed visibility, or update state partially.

Why it mattersAgents that retry on timeout duplicate real-world actions when the first call actually succeeded. Verifying the postcondition before retrying, plus idempotency keys, cuts duplicates without touching the model.

articleIsham Kalappurackal Mansoor, Abhishek Phadke, Pratip Rana

Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds

This paper presents a controlled study of self-evolving agent skill systems across five benchmarks and three models, finding that skill improvement is sparse (only 55 of 388 candidates beat their validation baseline) and depends critically on including failed trajectories as feedback, not just successes.

Why it mattersIf you're building self-evolving agent skill systems, this shows gains are sparse and depend on including failed trajectories as feedback.

articleYuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song

TabletCraft: Bridging a 4,000-Year Cultural Gap with Bidirectional Akkadian NMT and Cuneiform Rendering

An open-source toolkit that enables two-way translation between Akkadian and English plus rendering of composed English text as cuneiform clay tablets, using a ByT5 model trained on 116K bidirectional samples and a sign converter covering 95.3% of known cuneiform signs.

Why it mattersTabletCraft lets non-specialists compose English into cuneiform and translate Akkadian both ways via a ByT5 model, opening ancient-language work to builders outside Assyriology.

Evaluating OpenAI's Privacy Filter: Cross-Lingual, Cross-Domain PII Detection Across 42 Benchmarks

An independent evaluation of OpenAI's 1.5B-parameter Privacy Filter (OPF) across 42 synthetic benchmarks spanning 22 languages and 5 domains finds it outperforms Presidio and XLM-RoBERTa on structured PII but collapses on narrative prose and non-Latin scripts (e.g. Arabic F1=0.04).

Why it mattersOpenAI's Privacy Filter beats existing tools on structured PII like emails and phone numbers but fails badly on non-Latin scripts and narrative prose, while GPT-4o still leads on medical/legal/financial PII.

articleRohith Uppala

When Policies Change Probabilities: Modular Decision-Making for LLM Code Review

An empirical study of four LLM code-reviewer systems shows that embedding cost-sensitive approval policies directly in the risk-estimation prompt distorts the reported failure probabilities themselves, degrading decisions under high-cost settings to worse than rejecting every patch.

Why it mattersEmbedding cost-sensitive approval policies directly in a risk-estimation prompt can corrupt the reported probabilities themselves, sometimes making decisions worse than a blanket rejection under high-cost settings.

articleRasvik Kudum, Max Corbett, Hitansh Paliwal, Romaisa Fatima, Thomas Jiralerspong, Sneheel Sarangi

An index of the vibe-coding frontier. Corrections welcome.