Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster

Meta's account of Prometheus, its gigawatt-scale training cluster in New Albany, Ohio, is mostly about one problem: when a training job runs across tens of thousands of GPUs, a fault on a single host can slow or stop the whole run (engineering.fb.com). The first phase of the cluster is complete and trained Meta's Muse Wonder models. Meta is working toward roughly 1 million GPUs. It tracks reliability with two measures. Goodput is, roughly, the share of a job's time that turns into useful training progress. Mean time between interruptions (MTBI) is the average stretch, counted in GPU hours, between faults that halt a job. For the Muse Wonder runs, Meta reports more than 85% goodput in pre-training, more than 90% in reinforcement learning, and an MTBI of about 47k GPU hours. Later fleet changes, including firmware fixes, have since raised MTBI above 54K GPU hours.

The hardware made the problem harder. NVIDIA's NVL72 racks tie 72 GPUs into one tightly coupled group. The previous H100 generation coupled eight, and Meta says the larger group increases the blast radius of a fault by 9X. Meta's answer is to plan capacity around the whole rack. Each rack runs a 64-GPU slice, a power-of-two size that training parallelism maps onto efficiently, and keeps its other eight GPUs as in-rack spares. When a GPU fails, a spare takes its place and the job restarts from its last checkpoint instead of waiting for a repair. Meta also holds roughly 2% extra racks for rarer whole-rack faults. Together, these mean a 32,768-GPU run needs 523 racks, or 37,656 GPUs, so about 13% of the provisioned GPUs sit outside the active job. Meta notes that most of that margin, 11%, comes from the 64-of-72 slice geometry rather than from a reliability reserve.

Meta's interruption figures show how much of this work remained. During the cluster's initial phase in mid-2025, interruptions peaked at nearly 1,000 events per 8K GPUs a day. Meta says a year of engineering cut that by 99%, with fixes ranging from screening weak NVLink connections before production to automatically isolating faulty NVLink switch components. Its stated target is two interruptions per 8K GPUs per day, about 100k GPU hours MTBI, which the current 54K figure has not yet reached.