Reflection's Beam: strong on workflows, weak in the terminal
- Source
- x.com
- Date
Reflection’s first model, Beam, scores 30 on the Artificial Analysis Intelligence Index, placing it #2 among US open weights models Beam is an open weights model with 501B total and 23B active parameters, released on @huggingface Congratulations to @reflection_ai on the release! Key takeaways: ➤ Second among US open weights models: Beam scores 30, behind only Meta's Muse Spark Open (39) and ahead of yesterday’s NVIDIA Nemotron 3.5 Super VL (26) release. Outside China, only Muse Spark Open and the UAE's K2 Horizon (31) score higher ➤ Token efficient for its intelligence: Beam uses ~26k output tokens per Intelligence Index task compared to K2 Horizon, one point ahead, which uses twice as many (~52k). Proprietary models still set the frontier, with GPT-6 Astra (max) scoring 53 on the same budget (~27k) ➤ Strong on agentic workflows, weaker in the terminal: Beam scores 48.5% on AutomationBench-AA, 11 percentage points ahead of K2 Horizon (37.2%) and level with Qwen3.8 27B (48.2%). Terminal-Bench 4.0 is a weak point, where Beam scores 3.0%, behind Qwen3.8 27B (5.6%) and Muse Spark Open (8.1%) ➤ Knows more, guesses more: On AA-Omniscience it answers 33% of questions correctly, well ahead of K2 Horizon (18%) and Qwen3.8 27B (16%). But its hallucination rate is 47%, against 26% for K2 Horizon and 30% for Qwen3.8 27B, giving Beam a net index score of 1.5, ahead of both peers (−3.0 and −10.0) Other model details Context window: 1M tokens Modalities: Text Parameters: 501B / 23B
Beam scores 34% of on the Artificial Analysis Cyber Index, making it the leading US open weights model that we’ve tested Driven by its token efficiency, it also sits on the Cyber Index vs. Output Tokens per task frontier, alongside GPT-6 Sol (Daybreak Blue)
Beam uses ~26k output tokens per Intelligence Index task: ~19k for reasoning and ~7k for answers. Only 26% of its output goes to final answers, the 28th-lowest share of 219 models tested on Intelligence Index v4.3
Running in Mirror, Reflection's open-source CLI harness, Beam scores 28 on the Artificial Analysis Coding Agent Index. It sits 15 points behind the next setup, Claude Code with Qwen3.8 Max (43), and 40 behind the leaders (68). It is, however, frugal: only Codex with GPT-6 Astra and Codex with GPT-6.1 Sol use fewer tokens
Beam scores 30 on the Intelligence Index with about 26k output tokens per task and 48.5% on AutomationBench, but only 3.0% on Terminal-Bench and a 47% rate. That shows where to use it and where not to.
- Artificial Analysis scores Reflection's Beam (501B total, 23B active) at 30 on its Intelligence Index, second among US open-weights models behind Meta's Muse Spark Open at 39.
- Beam spends about 26k output tokens per Index task, half of K2 Horizon's 52k for one point less, and only 26% of that output is final answer text, with about 19k tokens going to reasoning.
- On AA-Omniscience Beam answers 33% correctly, well above K2 Horizon (18%), but hallucinates at 47% versus 26%, still netting an index score of 1.5 against K2 Horizon's -3.0.
- In Reflection's open-source Mirror , Beam scores 28 on the Coding Index, 15 points behind Claude Code with Qwen3.8 Max and 40 behind the leaders, though only two Codex setups use fewer tokens.
- Beam scores 34% on the Artificial Analysis Cyber Index, the highest of any US open-weights model it has tested, and sits on the cyber score versus output-tokens frontier alongside GPT-6 Sol.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…