AIMIP gives AI climate models a shared held-out benchmark
- Source
- x.com
- Date
Announcing the AI Model Intercomparison Project (AIMIP), a community effort to support open evaluation of AI climate models. 🌎 It brings together a shared benchmark experiment & dataset to make it easier to compare models side by side over multi-decade simulations. 🧵
We need transparent ways to evaluate how AI climate models perform on long-horizon forecasting. Weather models already have common evals like WeatherBench; AIMIP is a shared benchmark for AI climate modeling in the spirit of the Coupled Model Intercomparison Project (CMIP).
For AIMIP, models forecast the global atmosphere over 1979–2024, using historical data from 1979–2014 for training and leaving the final decade held out for testing. The benchmark focuses on the atmosphere alone, and leaves model architecture choices up to each submitter.
AIMIP evaluates model performance on: ◙ Overall climate averages ◙ Long-term trends ◙ El Niño-related atmospheric responses ◙ Day-to-day variability ◙ Out-of-sample behavior under warmer sea surface temperatures
Context
AIMIP gives AI climate models a common test of whether they reproduce both familiar climate patterns and subsequent warming. In the first phase, announced by Ai2, models trained on historical data from 1979–2014, leaving 2015–2024 for testing. Ocean and sea-ice conditions were supplied to the models, keeping the experiment focused on the atmosphere. The project description explains this shared setup.
Ai2 reports that the eight submitted models generally reproduced average historical climate well, while some substantially underestimated warming trends. Their responses also diverged under unfamiliar ocean-warming tests. These comparisons show why matching historical averages is insufficient evidence that a model will respond reliably to changing climate conditions.
Checking sign-in…
Loading comments…