AIMIP sets a shared held-out benchmark for AI climate models

AIMIP, the AI Model Intercomparison Project, gives AI climate models one shared experiment so their results can be compared directly. Ai2 leads the effort, and its announcement explains the gap it fills. AI weather models are already judged on common benchmarks for forecasts one to ten days ahead, but climate models simulate decades, and the existing intercomparison projects were designed for conventional physics-based models. Without a common setup, two AI climate models trained and tested in different ways cannot be lined up fairly.

In Phase 1, each group simulates the global atmosphere from 1979 to 2024 with an architecture of its choice, but may train only on ERA5, a reconstruction of past weather from observations, for 1979 to 2014. The last decade is held out, so the test asks whether a model captures warming it never saw in training. Observed ocean surface temperatures and sea ice are supplied as inputs, which keeps this phase about the atmosphere alone. Models are scored on average biases, the historical warming trend, the response to El Niño, day-to-day variability, and an out-of-sample test in which the global ocean surface is instantly warmed by 2 or 4 degrees Celsius.

Eight simulations came from Ai2 and five outside groups, including NVIDIA and Google Research. The Phase 1 paper finds the AI models simulated the historical climate about as well as a conventional physics-based model, and Ai2's post says they typically beat the conventional GFDL-CM4 model at reproducing average patterns. The weak points appeared where conditions changed. Some models significantly underestimated the warming trend in the held-out decade, and under the sudden ocean-warming test their predictions diverged, with some producing what the authors call apparently physically implausible results. Ai2's post concludes that AI models are competitive on key climate measures but struggle to respond robustly across a range of scenarios, and says this ability to generalize will be critical for wider adoption. The Phase 1 dataset is public, with code and evaluations on GitHub.