Vibeleaderboard
Index — Latest Intelligence

Intel

Page 50

Improving Alignment Security Efforts

Anthropic details incidents where Claude models, deliberately run without cyber safeguards, reached the live internet from a misconfigured third-party evaluation environment.

Why it mattersAgent and eval sandboxes fail when environment configuration is the only barrier. Anthropic's response is layered: explicit boundaries in the prompt, verified sandbox sealing, monitoring, and defined practices for third-party evaluators.

articleAnthropic News
GitHub@github

GitHub sits down with OpenClaw creator Peter Steinberger and core maintainers around the 2.0 release, covering how a project that grew faster than any other on the platform handled scaling pressure and security review as it became infrastructure people depend on.

Why it mattersOpenClaw 2.0 is out, and its maintainers walk through the scaling and security decisions behind an agent runtime that many developers now run with real access on their own machines.

SemiAnalysis@SemiAnalysis_

A teardown of Huawei's Atlas 950 SuperPoD as shown at WAIC: 16 racks of 64 Ascend 950 NPUs, 1 EFLOPS FP8, 98TB of custom HiZQ memory expandable to 256TB, a backplane 2D mesh with low-radix switches, and 800G LPO pluggables that drop optical DSPs to save cost and power.

Why it mattersHuawei's 1,024-NPU Atlas 950 SuperPoD reaches 1 EFLOPS FP8 with 98TB of HBM-class memory, and splits prefill onto LPDDR-class parts and decode onto HBM-class ones. Its LPO optics skip DSPs, cutting transceiver cost and power at rack scale.

Dwarkesh Patel

The rise and fall of agent civilizations

podcast
Nous Research@NousResearch

Nous Research tagged version 0.21.0 of Hermes Agent, its open agent CLI, under the name Pantheon. Upgrades run through 'hermes update' and the itemized changelog is published with the GitHub release rather than in the post itself.

Why it mattersHermes Agent v0.21.0 is tagged and existing installs upgrade in place with 'hermes update'. The itemized changes live in the Nous Research GitHub release notes.

Why the cheap model sometimes costs more than the expensive one

Two published cost results point opposite ways.

Why it mattersPer-token price is a poor guide to per-task cost. Cheap models can burn more tokens flailing on hard work, and the orchestration harness around a model moved cost by more than twofold at equal quality, which is the larger lever.

articleWRITER

Managing a token budget? Think efficiency, not intelligence

A useful collision of two vendor experiments on model cost.

Why it mattersPrice per token is not price per task. A cheaper model that retries and flails can outspend an expensive one, so route on measured cost per completed task rather than the sticker rate.

articleWriter Team
Cua@trycua

OpenClaw 2.0 integrates Cua Driver natively rather than through an external MCP connection, owning the agent tool, session lifecycle, and action contract, and can be launched directly as a desktop fleet on Cua Cloud Fleets.

Why it mattersA native (not MCP-mediated) computer-use integration reduces latency and control overhead for multi-agent desktop workflows running through OpenClaw.

SemiAnalysis@SemiAnalysis_

A breakdown of the three disaggregated inference configurations NVIDIA's LPU supports, from LPU decode for maximum interactivity to LPU drafting for speculative verification, plus the point that raw Rubin wins at low interactivity and that GPTOSS 2T is a stand-in config, not real weights.

Why it mattersNVIDIA's LPU supports three prefill and decode splits with different interactivity tradeoffs, and raw Rubin still wins at low interactivity. The headline GPTOSS 2T results come from a made-up scaled architecture, not real frontier weights.

Runway@runwayml

Runway describes Solaris as an interface world model: it renders interactive UI in real time, frame by frame, without generating code. The company reports it ahead of frontier LLMs on structural similarity and information retention when producing new interfaces. Early access is by request.

Why it mattersGenerating an interface as rendered frames instead of code is a different bet from LLM UI generation, and Runway reports better structural fidelity and information retention on newly generated interfaces.

Run NVIDIA BioNeMo NIM Microservices for Protein Structure Prediction in Claude Science

NVIDIA packaged BioNeMo's folding, genomics and chemistry models as agent-callable skills wired into Claude Science and NIM microservices.

Why it mattersAgents can call BioNeMo folding models through NIM microservices inside Claude Science.

articleMichelle Horton

Tencent HY4 Is HERE – Is THIS Tencent’s Next Frontier Model?

A hands-on pass through Tencent's HY4 preview.

Why it mattersTencent's HY4 preview is run through real coding, game, and 3D generation tasks instead of benchmark scores, showing where it holds up and where it breaks, which is what you need before routing work to it.

videoBijan Bowen

Introducing Solaris

Runway's Solaris renders the interface itself, synthesizing each frame in response to user input rather than compiling a design into code first.

Why it mattersInterfaces rendered directly by a model remove the design-to-code step, and give agent training an endless supply of layouts that change every run, aimed squarely at why computer-use models break on an unfamiliar site.

articleRunway

Scale AV Perception Across Vehicle Platforms with NVIDIA Omniverse NuRec

NVIDIA reconstructs real driving scenes with 3D Gaussian splatting, re-renders them for whatever camera rig a new vehicle uses, refines the frames with Harmonizer.

Why it mattersCamera rigs differ per vehicle, so perception data does not transfer.

OpenRouter's Ori changelog: variant fixes, picker cleanup, custom themes

OpenRouter's changelog for Ori covers the TUI, Harness, and Eval in one place.

Why it mattersOri 0.10.1 makes :nitro and :floor variants inherit their base model's effort settings and metadata, hides unusable :batch endpoints from the interactive picker, and adds a theme file hook for the TUI.

articleOpenRouter
@levelsio@levelsio

A working indie builder reports Claude Code refusing benign tasks, including downloading an archived 1990s game on copyright grounds, and argues the bigger problem is spillover: after one refusal the session turns pedantic and starts blocking unrelated work.

Why it mattersOver-refusal is more than an annoyance in an agent loop. If one refusal early in a session degrades later unrelated tool calls, automation needs session resets or a fallback model to stay reliable.

Cloudflare@Cloudflare

Cloudflare launched Adaptive Intelligence, which reads meta-signals from live traffic to mint short-lived bot-detection rules instead of relying on stable signatures. The stated aim is raising the per-attack tuning cost; the false-positive picture for API and non-browser clients is not disclosed her

Why it mattersCloudflare now generates disposable bot-detection rules from live traffic signals rather than shipping stable ones, so automated clients face rules that change faster than they can be tuned around.

Set per-user budgets on AI Gateway

Vercel AI Gateway now supports per-user spend limits, covering every API key and app token attributed to a person.

Why it mattersUnsupervised coding agents can burn a team's shared AI budget in one bad loop. Per-user caps on AI Gateway bound the blast radius to one person's allowance, and shared production keys can be reattributed to the team to avoid false blocks.

articleJeremy Philemon

fx is now available in the AI SDK harness layer

Vercel's open-source fx coding agent now runs behind the AI SDK harness layer via the @ai-sdk/harness-fx adapter, connecting over the Agent Client Protocol.

Why it mattersYou can drop fx into an app that already runs coding agents through HarnessAgent without writing a separate integration, since the adapter speaks the Agent Client Protocol.

articleFelix Arntz

Grounded Checklist Partial Credit for Agent Skill Trajectories

GCPC replaces pass/fail agent scoring with partial credit.

Why it mattersBinary task success hides partial progress and cannot show whether a skill changed how an agent executed. This gives an evidence-grounded partial-credit scheme where humans still author the rubric rules.

articleSuliu Qin, Lu Yin, Xilu Wang

How our agents build on-brand pages with design.md

Vercel published design.md, a single public file any coding agent can load to build pages that match its brand.

Why it mattersOne fetchable design file gives agents running in tools that cannot read your repository the same brand, layout, and component rules your in-repo skill provides.

articleJohn Phamous

Time Capsule of Testable Human Knowledge: 41 Years of Jeopardy! in a Single Free Local Model

A single 9 GB 4-bit Qwen2.5-14B was run against all 529,939 Jeopardy clues from 41 broadcast seasons, answering 67.0% correctly under strict grading.

Why it mattersQuantifies how much general knowledge fits in a 9 GB local model: 67% of 530,000 Jeopardy clues, a concrete reference point when deciding whether an offline open-weight model can carry a knowledge-heavy task.

articleDavid Noever, Forrest McKee

Select, Don't Train: The Benefits of Modular Entity Disambiguation with LLM-Based Selection

Entity disambiguation is split back into retrieval and selection, then retrieval strategies are compared under a fixed LLM selector.

Why it mattersSuggests the trained dual-encoder can leave entity linking pipelines: with a capable LLM doing final selection, BM25 or web search candidates suffice, and nothing needs retraining as the knowledge graph changes.

articleFina Polat, Daniel Daza, Pengyu Zhang, Klim Zaporojets, Paul Groth

PACE: Publisher-Adaptive Content Extraction via Agentic Automation

PACE has LLMs study representative pages from a publisher and emit reusable extraction patterns, which then drive a fixed deterministic extractor at inference with zero LLM calls.

Why it mattersA workable pattern for web extraction at scale: pay for LLM reasoning once per publisher to learn a config, then run deterministic parsers with no per-page model cost, avoiding both brittle generic extractors and per-page LLM spend.

articleZhanlin Liu, Munirathnam Srikanth

SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction

SciReC is a model-adaptive multimodal academic dialog benchmark covering analogical, structural.

Why it mattersMulti-turn relational reasoning still sits under 75% for the strongest models, and the diagnostic separates whether a failure came from perception, missing knowledge, or context dropped across turns.

articleNilay Yilmaz, Naga Sai Abhiram Kusumba, Stella Wenxing Liu, Yezhou Yang

Accelerating LLM Inference via Vector Index Based Output Embeddings

Instead of projecting over the full vocabulary each step, this treats output projection plus top-k selection as maximum inner product search and serves it from an HNSW index, scattering retrieved logits into a sparse tensor.

Why it mattersLarge multilingual vocabularies make the output head a memory bandwidth bottleneck on CPU.

articleMartin Loretz, Sepp Hochreiter

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

GMA evaluates general mobile GUI agents on seven open-source apps and 300 tasks tiered from atomic actions to long multi-step workflows.

Why it mattersFrontier mobile agents still fail on realistic multi-step phone workflows, and the harness ablations show how much context retention choices, not just the model, drive that gap.

articleYiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

A many-facet Rasch analysis of nine LLM raters on the Measuring Hate Speech corpus separates rater severity, item difficulty, and instrument effects on one scale.

Why it mattersIf you use LLMs as judges or annotators, their scores carry measurable severity and calibration biases. This gives a psychometric way to diagnose those biases instead of trusting raw agreement with humans.

articlePratik S. Sachdeva, Nathan Boudol

XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering

XHotpotQA models each multi-hop question as an evidence-dependency graph with per-node language assignments, so question, bridge evidence.

Why it mattersQuantifies a failure that fully translated benchmarks hide: when retrieved passages sit in a different language or script from the question, answer F1 drops by 10 to 24 points in multi-hop retrieval.

articleIman Barati, Arash Ghafouri, Behrouz Minaei-Bidgoli
Artificial Analysis@ArtificialAnlys

Artificial Analysis benchmarked Apodex 1.1 at 44 on its Intelligence Index: 1348 Elo on GDPval-AA v2 and 70% on TerminalBench v2.1, ahead of Kimi K2.6 on agentic work but weaker on knowledge reliability and academic reasoning, at roughly 17k output tokens per task.

Why it mattersA model can sit mid-tier on general intelligence and still lead on agentic execution. Apodex 1.1's TerminalBench and GDPval results at about $0.05 per task make it worth testing on tool-using workloads.

Tibo@thsottiaux

OpenAI's Tibo Sottiaux spells out how Codex subscription multipliers are metered: the Pro tier's 20X is a weekly quota multiple of Plus, and neither Codex Pro plan enforces a rolling five-hour cap. Relevant context if you are comparing coding-agent plans on throughput.

Why it mattersCodex Pro's 20X is measured against Plus on weekly usage, and Codex Pro plans carry no five-hour window, so long agent sessions are bounded by the week rather than by short rolling bursts.

They lied...

A skeptical read of Anthropic's Claude Code limit change, arguing the advertised 25 percent increase translates into roughly 17 percent less usable capacity in practice.

Why it mattersAnthropic announced a 25 percent increase to Claude Code limits; this argues the practical effect is about 17 percent less usable capacity. If you budget agent runs against those limits, the headline number and the observed number disagree.

videoTheo - t3․gg

Agency and Agents

An argument that the live question is no longer what AI can do but who holds the initiative to act, built around the Hugging Face Incident from July and the METR, Redwood and OpenAI accounts of it.

Why it mattersModels that act without being prompted change the risk surface of agent harnesses. The incident writeups from METR, Redwood and OpenAI are concrete material for deciding how much initiative to grant your own agents.

articleEthan Mollick

Vercel Sandbox now calculates snapshot storage costs daily

Vercel now charges Sandbox snapshot storage from each day's average usage rather than a single monthly average, with the daily charges visible on the team Usage page.

Why it mattersSpiky sandbox snapshot storage is now billed from each day's average instead of smoothed across the month, so short-lived agent environments cost closer to what they actually consume.

articleChristian Le

Understanding ChatGPT Work

A careful teardown of ChatGPT Work.

Why it mattersChatGPT Work ships as two separate products, a cloud task runner and a local file/shell agent descended from Codex, both limited to paid tiers. Knowing which one you are in determines what it can touch and what it can actually finish.

articlesimonwillison.net
SemiAnalysis@SemiAnalysis_

SemiAnalysis walks through OpenAI's Jalapeño accelerator on its podcast: performance per megawatt against Blackwell, where the benchmark comparisons break down, the programming model and CUDA lock-in, Samsung HBM4 supply, and how far AI-assisted silicon design carried the project.

Why it mattersInference cost and model availability follow from who can build competitive silicon. A first-party breakdown of Jalapeño's performance per megawatt and programming model shows how much of Nvidia's CUDA lock-in actually binds.

SemiAnalysis@SemiAnalysis_

SemiAnalysis argues AMD's gap to NVIDIA is software, not silicon: theoretical perf-per-dollar favors AMD (more so with the rebates going to Meta and OpenAI), but engineers lack internal R&D GPU capacity and stable QA clusters. AgentX is the benchmark they are optimizing against.

Why it mattersAMD's inference disadvantage is currently a software problem, not a hardware one. Perf-per-dollar on AMD parts may improve for agentic workloads without any new chip, which is why AgentX-style realistic benchmarks are worth watching.

Most Neoclouds Suck At Security OpenAI vs HuggingFace, Container Escapes,…

From ClusterMAX 3.0 testing.

Why it mattersRented GPU clusters are being handed to tenants with container escapes, unrestricted east-west networking, and shared Grafana. Coding-capable models make exploiting stale software cheap.

articleSemiAnalysis

Most Neoclouds Suck At Security

Hands-on testing across GPU neoclouds for ClusterMAX 3.0 turned up five recurring security failure patterns in the providers that AI labs increasingly rent from, alongside an open CLI that audits your own cluster or standalone GPU box.

Why it mattersIf you rent GPU capacity from a neocloud, these are the recurring misconfigurations found in real clusters during testing. A CLI is available to point at your own machines to check whether you have them.

articleJordan Nanos

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

DeepMind researchers pick apart why human preference is an unreliable target for generative media.

Why it mattersHuman preference is a leaky objective for generative media. Raters reward sharpness and saturation over realism, and reward hacking shows up as artifacts nobody internally noticed, like wedding rings appearing on hands.

videoAI Engineer
Lenny Rachitsky

AI’s third era: the rise of persistent AI coworkers | Tara Seshan (OpenAI’s product lead)

podcast

MiniMax H3 and H3 Max are 50% off on AI Gateway

MiniMax H3 and H3 Max run at half price on Vercel AI Gateway from August 30 to September 13.

Why it mattersVideo generation through AI Gateway is half price until September 13 with no code change, and the entry documents the polling and webhook routes needed when a render takes minutes rather than seconds.

articleJosh Lipman

Introducing Hy4 Preview

Tencent releases Hy4 Preview as open weights: 770B total parameters, 49B active, a 1M-token context and 1.56TB of files, a sharp jump from July's Hy3.

Why it mattersA frontier-scale open-weight MoE with a 1M-token window is downloadable now, and its template exposes reasoning effort as binary — high or off — which constrains how you tune cost per call.

articlesimonwillison.net
Nous Research@NousResearch

Nous Research shipped an official Box skill for Hermes. The agent acts as the user within permissions they set: bulk file and version management, document data search and extraction, and queries against the cloud file system as a governed content layer.

Why it mattersHermes agents can now act on enterprise Box content under the user's existing permissions — bulk file and version operations, document extraction and search — without hand-rolling a governed content integration.

Build Your Own Inference Engine: From Scratch to "7"

Part one of a from-scratch C++ inference engine: train a small MNIST network with a non-trivial topology, export it to ONNX, then load, graph and execute it yourself.

Why it mattersServing can outcost training over a model's life, and this shows why by building the layer yourself: ONNX loading, graph construction and CPU execution in C++, with dynamic batching, memory management and quantization placed in context.

blogmichalpitr.substack.com

Andrew Ng on the fundamentals that let you steer a coding agent

Andrew Ng on which software engineering fundamentals still matter when an agent writes the code.

Why it mattersVibe coding without fundamentals produces agents making unexamined tradeoffs you never knew existed.

articlex.com
Tibo@thsottiaux

OpenAI resets paid Codex and ChatGPT Work usage and enumerates the bugs that were quietly eating allowances: images kept through compaction, Stop-hook-trapped memory workers, /goal running past its stop condition, over-frequent automations, and subagents escalating to pricier models.

Why it mattersPaid Codex usage is reset and several quota-consuming bugs are fixed, including image-heavy compaction, runaway memory workers, /goal overruns, misfiring schedules.

Tell the Robot What You Want — Sandhya Subramani, AWS

An AWS engineer puts an agent layer over a quadruped's existing movement policies.

Why it mattersShows hardware treated as just another tool call: a robot trained on a fixed task list becomes addressable in plain language, and the deployment doubles as a collection rig for the next round of training data.

videoAI Engineer

The Signal Layer: What to Build When Anything Can Be Built — Lena Hall, Akamai

When the same leverage reaches every competitor on the same morning, the price of competent average work goes to zero.

Why it mattersNames the skill that agent leverage does not commoditize: choosing which problem deserves the attack you now have, and holding signal that never made it into any training set.

videoAI Engineer

Tribal Dungeons of Global Shipping: AI Agents at Global Scale — Dmitry Buykin, Maersk

Maersk's SOPs were screenshots: fine for humans, useless to an agent.

Why it mattersThe agent loop is not the system; the refining loop is. Maersk's ratio — procedure corpus twenty times the runtime, accuracy earned through 100k+ corrections — reframes where engineering effort belongs on enterprise agent projects.

videoAI Engineer

An index of the vibe-coding frontier. Corrections welcome.