Ai2

Ai2 details how OlmoEarth Run splits continental inference jobs into independent windows across CPU prep, GPU inference and CPU postprocess stages, with idempotent tasks and an imagery index. A North America wildfire run cut 4,737 serial hours to 30.5.

Ai2's Skylight team outlines how Shippy, a maritime intelligence agent, earns analyst trust: a typed CLI instead of shell access, large results written to disk, isolated ephemeral sessions per conversation, and expert-written evals scored by an LLM judge that block regressing builds.

Ai2's infrastructure team explains replacing a priority scheduler with GPU time budgets, hierarchical fair-share allocation and a time-slicing contract, aiming to raise the impact of scheduling on clusters of 88 to 1024 H100, B200 and B300 GPUs.

Ai2 describes Steering Arena, a crowdsourced game scoring prompts by how strongly they shift Olmo 3-32B toward a prosocial internal pattern. The top 36 of about 600 entries were nonsense strings, and one player used automated search, showing how an exposed metric becomes an optimization target.

A national language-model effort in Denmark builds on FlexOlmo's modular experts to sidestep the data-sharing wall: regulated and proprietary corpora stay put, independently trained modules get merged into a shared model. A working template for federated model building under legal constraints.

Ai2's FairStress benchmark tests whether models hold correct and tied decisions equally firmly across demographic groups. It finds asymmetric robustness and different explanation language, and that most of nine debiasing methods fail to help. Data and code are open, with 300 decisions across 29 doma

Ai2 releases AstaBrief 8B, a Qwen3-8B fine-tune (SFT plus DPO) that turns research questions and literature excerpts into cited reports. Weights and training data are open, and it powers Fast mode in Asta, running about 3.5x faster than the Claude-based mode.

Ai2's BenchMIRT applies multidimensional item response theory to 100 models across 16 benchmarks and 34K prompts, estimating each question's difficulty and which latent ability it discriminates. Without label supervision it recovered safety and general reasoning as the dominant dimensions.

Ai2's infini-gram indexes enormous text corpora and counts phrases of any length, which turns 'is this AI-written' into a checkable question: which expressions in a passage also appear in prior published work. Applied here to research on AI-generated books and their overlap with existing titles.

Goodfire used Ai2's open post-training stack to trace a safety regression back to specific preference-training examples, then tested targeted fixes without sacrificing unrelated capability gains.

Ai2's maritime agent had to be right or a patrol vessel goes the wrong way. The team's answer: treat the agent as three separable things, prompt persona, skills, and runtime config, pin the first two into a versioned image, and validate against live satellite and vessel feeds.

Ai2 introduces Old Ideas, Novel Problems, a study and benchmark measuring how reliably LLMs judge research-idea novelty. It ships data, code and a pipeline for building new evaluation sets for novelty judges.

Ai2 turned a prosocial-steering evaluation into a public game, Steering Arena, where players submit text prefixes and see how strongly they shift Olmo 3's responses, using the model's open internals rather than just its outputs.

Benchmark averages hide where an architecture wins. Ai2 held data, tokenizer, and recipe constant across a 7B transformer and its hybrid twin, then compared predictions token by token, finding the hybrid's edge concentrated on meaning-carrying tokens and those needing long-range context.

Ai2 releases Olmo-core 3, a redesigned open training stack for large mixture-of-experts models targeting trillion-parameter scale. It tackles expert routing and communication overhead, with code, a tech report and an interactive demo.

Existing MoEs route sparsely but still need the whole model resident. Ai2's EMO is pretrained so modular structure emerges from the data itself, letting a task run on 12.5% of the experts at close to full-model quality while the complete set still behaves as a general-purpose model.

A side project stacks Molmo, MolmoWeb and Olmo 3 into an accessibility checker that judges what a page actually renders rather than what its HTML claims. The write-up is worth reading for the pattern: using a vision model as the oracle where static rule engines are structurally blind.

Researchers separate what a model knows about a drug from what it can infer from the drug's name, then trace the shortcut back into the training corpus. The numbers are blunt: over half the drugs tested showed little sign of specific knowledge underneath a fluent answer.

Ai2 introduces StateBench, a benchmark that grades how well language models follow entity state changes across long documents, from simple latest-value lookups to multi-step chains, with data and code released.

Goodfire used Ai2's open Olmo 3 stack (Dolci preference pairs, checkpoints, OLMES evals) to predict which behaviors DPO will strengthen before training. In one run, capability gains came with more compliance on fictional-framed harmful requests, traced to specific preference examples.

Ai2 open-sources the evaluation workbench it uses while models are still moving, not after they ship. The framing is the useful part: benchmark suites designed for released models fail at reconfiguring tasks per intervention and tracking whether small-scale gains hold at scale.

A Georgia Tech pair traces where a model's social reasoning comes from by scoring how much each training document shaped specific benchmark answers. The method needs public corpora and intermediate checkpoints, making it a sharp illustration of what open weights alone cannot buy you.

Ai2 introduces FairStress, a benchmark probing whether LLM decisions stay demographically fair when challenged with persuasive arguments, not just whether the initial choice matches. Data and code are open, and it is being added to Olmo-eval.

Ai2's TutorMoments benchmark runs frontier models through 520 teacher-annotated tutoring moments. Under a plain prompt models over-help and rarely push for rigor. A prompt that spells out the trade-off improves every model, yet none adapts its moves to the situation as much as human tutors do.