Meta’s Prometheus AI Supercluster: GPUs, Networking and Reliability at Scale

Prometheus shows what changes when AI training infrastructure is designed as one large system rather than a collection of servers. Meta’s New Albany, Ohio, site now brings together hundreds of thousands of GPUs across multiple buildings, linked by dedicated networks and engineered to keep long-running training jobs productive when components fail.

The first phase is complete and was used to train Meta’s Muse foundational models, including Muse Wonder. Meta says it is now expanding the cluster with future models from Meta Superintelligence Labs in mind, with a long-term target of roughly one million GPUs. That ambition makes Prometheus a useful case study in the practical work behind large-scale training: coordinating hardware generations, moving traffic across buildings, managing power swings, and limiting the cost of faults.

Two years ago, the New Albany site ran conventional web workloads. Meta has since adapted it for model training through changes to hardware, networking and building design. The company describes the resulting system as a supercluster: separate data center buildings and rapid-deployment structures operate as one connected compute environment.

Meta’s video overview of Prometheus

The Prometheus supercluster is housed in Meta’s New Albany, Ohio, data center.

Training performance depends on the whole system

For a large training run, peak accelerator speed is only part of the picture. A failure on one host can stall a job involving many thousands of GPUs, reducing the useful work completed over time. Meta calls this measure “goodput”: the share of elapsed training time spent making productive progress.

During Muse Wonder training, Meta reports more than 85% goodput for pre-training runs and more than 90% for reinforcement learning runs. It also reports a mean time between interruptions of about 47,000 GPU hours across those runs. Subsequent firmware changes, the company says, raised that figure above 54,000 GPU hours. These are operational measures, rather than peak hardware specifications, and they help show why reliability work matters to model development schedules.

Prometheus is intended to support larger models and longer context windows than were practical on earlier systems. Its capacity is distributed across buildings, but training software and networking must keep the accelerators coordinated closely enough to behave like a single machine. The engineering challenge is therefore not simply adding GPUs. Compute, memory, fabric bandwidth, storage, cooling and power delivery all need to scale together.

Three accelerator platforms, with different tradeoffs

Prometheus currently uses two NVIDIA platforms, and Meta says a third, based on Vera Rubin NVL72, is expected to arrive. Each uses an NVL72 design in which 72 GPUs share a high-bandwidth network fabric. The first two deployments illustrate how Meta has adjusted rack layout and networking as the hardware has evolved.

Catalina: GB200 NVL72

Catalina is Meta’s first platform based on NVIDIA GB200 NVL72. Its Ariel configuration spreads 72 GPUs across two racks, with 36 GPUs in each rack connected through NVLink. The system uses Meta’s Open Rack V3 high-power rack design, an open hardware contribution to the Open Compute Project.

Meta chose Ariel to address an imbalance in the GB200 system’s networking. The platform increased compute, high-bandwidth memory capacity, internal bandwidth and the size of the tightly connected GPU group compared with the earlier H100 design. But Meta expected the external connection to the wider cluster, at 400G per GPU, to constrain performance. Ariel pairs each Blackwell GPU with a Grace CPU and two ConnectX-7 400G network cards, providing 800G of external network capacity per GPU. Meta says that configuration also gave it a consistent specification as faster networking became available across the region.

The published Catalina specifications are 72 Blackwell GPUs and 13.4 TB of HBM3e, or 192 GB per GPU. Meta lists peak compute at 720 PFLOPS for FP8 training and 1,440 PFLOPS for FP4, along with 130 TB/s of peak NVLink bandwidth and 576 TB/s of aggregate memory bandwidth. The two-rack system draws about 120 kW.

Clemente: GB300 NVL72

Clemente moves to the GB300 NVL72 and changes the physical layout as well as the GPU. HBM capacity rises from 192 GB to 288 GB per GPU, a 50% increase, while dense FP4 throughput rises by a similar factor. Meta also reports roughly twice the attention, or softmax, acceleration compared with GB200.

Networking is built into the platform at 800G per GPU through ConnectX-8 SuperNICs. That makes the faster external link a standard feature, rather than a configuration Meta had to add early to Catalina. Clemente also uses two GPUs per CPU, instead of Ariel’s one-to-one ratio, allowing all 72 GPUs to fit in one rack. The consolidated layout may simplify deployment and reduce cabling, while Meta says serviceability improves.

The rack can be configured with modular Air-Assisted Liquid Cooling or with Facility Liquid Cooling in newer data centers. Meta lists more than one exaFLOP of peak FP4 compute, 130 TB/s of peak NVLink bandwidth and roughly 135 kW of rack power. Its stated performance improvement over GB200 NVL72 is 5% to 10%.

A network built for synchronized training

Web service networks generally move requests and responses among machines that can operate with some independence. Training large models creates a different pattern: groups of GPUs exchange substantial amounts of data and often need to stay closely synchronized. If communication slows, accelerators can sit idle while waiting for data.

Meta added a dedicated Ethernet Back-End network alongside the network used for ordinary data center traffic. It connects GPUs both within a building and across buildings. The design reflects the way training work is divided. A job can create data-parallel model replicas; communication that is latency-sensitive and bandwidth-heavy tends to stay within a replica, while less time-sensitive traffic can move between replicas. Prometheus uses fast, low-latency fabrics inside buildings and an aggregation network to connect those building-level systems.

Meta says it works with Meta Superintelligence Labs to align parallelism choices, traffic patterns and communication software with network behavior. That coordination is meant to reduce the time GPUs spend waiting for exchanges. The company has applied related approaches in earlier clusters of 24,000 GPUs and more than 100,000 GPUs, then extended them to Prometheus’s greater distance and scale.

Building fabrics and cross-site links

Each building-level Ethernet fabric can connect up to 20,736 GPUs. Meta’s initial design used a Disaggregated Scheduled Fabric, or DSF, to handle a training traffic pattern that can produce a small number of large, long-lived flows. Such flows may collide under conventional load balancing. DSF addresses that behavior with routing at sub-packet granularity, packet reordering, virtual output queues and credit-based flow control.

As network cards gained the ability to reorder packets at the endpoint, Meta says it could shift to a Non-Scheduled Fabric, or NSF. That design uses shallow-buffer 51.2T Ethernet switches, including Minipack3 with Dynamic Load Balancing and Minipack3N, which uses NVIDIA Spectrum-4 switch silicon and Adaptive Routing on ConnectX network cards. Both fabric designs run on Meta’s FBOSS networking stack, which gives its engineers control over configuration and visibility into operation.

Building fabrics connect through Backend Aggregation, a multi-layer hub-and-spoke network spanning distances from 2 km to more than 100 km. It uses Arista 7800-series chassis with deep-buffered Broadcom Jericho3 silicon. Meta reports multiple petabits per second of bandwidth between regional pairs. The distance introduces a physical limit on how quickly data can travel, so training software and network behavior need joint tuning to prevent communication delay from limiting progress.

Meta’s explanation of Backend Aggregation

The full network includes millions of miles of fiber strand, tens of thousands of switches and millions of optical components, according to Meta. Shorter in-building links use 2×400G FR4 LITE pluggable optics; links beyond about 10 km use 800G-ZR coherent optics. This mix reflects the different cost and reach demands of connections within a building and between distant facilities.

Making faults less costly

The larger scale-up domain in NVL72 improves internal communication, but it also increases the potential impact of a fault. H100 systems used NVL8 groups of eight tightly coupled GPUs; NVL72 expands that group to 72. Meta describes the increase in potential fault scope as ninefold, which changes how it allocates capacity and plans recovery.

Rather than use all 72 GPUs in a rack as active participants, Meta assigns 64 to a training slice and keeps eight as in-rack spares. The 64-GPU group suits power-of-two training parallelism. If a GPU fails, a spare can take its place, allowing the job to resume from its latest checkpoint while the rack remains part of the run. Meta also holds about 2% extra racks for less common failures that affect an entire rack or pod.

For a training run with 32,768 active GPUs, Meta gives an example of 523 NVL72 racks, or 37,656 GPUs, about 13% more than the active count. Most of that extra capacity, 11%, comes from reserving eight GPUs in each 72-GPU domain; the remainder covers rack-level failures. Capacity planning therefore needs to account for both hardware redundancy and the way training jobs divide work.

The physical connections add another reliability challenge. A single NVL72 domain contains 5,184 tuned differential-pair signaling links operating at 200 Gbps. Each link passes through several components, including GPU SerDes, compute-tray circuit boards, cable cartridges and switch interfaces. A small signal-quality issue can matter when multiplied across such a dense fabric.

Meta describes a joint engineering process with NVIDIA that includes teams at the data center, shared telemetry and recurring fault analysis. Meta focuses on scheduling and infrastructure; NVIDIA works on firmware, signal diagnostics and memory handling. The aim is to identify the component and failure mechanism behind each halt, then assign a targeted fix.

Recovery paths and interruption reduction

When a fault is limited to one data-parallel replica, Parallelism-Aware Fault Tolerance can take that replica offline while other replicas repeat the affected step and continue training. The failed portion can restart on spare hosts and synchronize with the rest of the job. If the fault reaches beyond one replica, the full job may need to restart from a checkpoint. Meta’s distributed checkpoint system saves model parameters, gradients and optimizer data asynchronously to multi-tier storage, using delta checkpoints and in-memory staging to limit the effect on training throughput.

These techniques work alongside spare GPU trays and reserve racks. The goal is to reduce both the time spent waiting for replacement hardware and the amount of completed work lost when a job must recover.

Meta says Prometheus experienced nearly 1,000 interruptions per day per 8,000 GPUs during its initial phase in mid-2025. After a year of engineering work, the company reports a 99% reduction, enabling several training jobs involving tens of thousands of GPUs to run in parallel. Its stated target is two interruptions per 8,000 GPUs per day, which it equates to about 100,000 GPU hours between interruptions.

The reported improvements came from work on several failure categories. For GB200, NVLink signal integrity was an early bottleneck. Meta monitors connections and uses L11 diagnostic burns to flag weak links before production. Its tools distinguish transient faults from persistent signal problems, after which systems can be repaired and checked before returning to use.

As GB300 deployments added memory capacity, HBM and LPDDR faults became more prominent. Meta says it has worked with NVIDIA to analyze fleet telemetry and adjust HBM firmware settings, memory scrubbing and LPDDR error correction. Pattern-based detection is also used to identify degrading devices before they slow training. Meta says these changes are being rolled out and their effect on interruption rates and replacement needs is still being measured.

Switch and fabric issues receive millisecond-level monitoring, with automated recovery intended to isolate a faulty component and shift traffic without stopping a job. At the Backend Aggregation super-spine, hardware fault detection and planar network layouts allow traffic to route around individual fiber or port failures.

Operations matter alongside the design. Meta points to a triage team based at the New Albany site, a stability sprint in March that coordinated fixes across the system, and a rule that repaired hardware must pass a quarantine period before rejoining the compute pool.

Fleet operations, power and cooling

Hardware behavior at large deployment volumes can reveal faults that are difficult to spot in smaller installations. Meta says it built automation to apply firmware and driver updates to more than 100,000 GPUs within hours. The 90th-percentile time for a single-pod upgrade fell from more than seven hours early in the hardware rollout to two hours.

Other fleet practices include correlating telemetry across racks, screening more than 600,000 NVLink paths with stress tests, qualifying hardware with multiple manufacturing partners and testing power supplies and voltage regulators under the 1,400 W load of individual GPUs. The company also checks cold plates and manifolds, keeps replacement parts ready, and runs burn-in and pre-job health checks to select nodes for large jobs. Meta says lessons from Blackwell operations will inform its Vera Rubin deployment.

Power demand is another system-level problem. During synchronized training, thousands of GPUs can enter compute-heavy phases together, producing cluster load swings of 30% to 40%. Starting or pausing a large job can shift demand by tens of megawatts within seconds. Meta’s Dynamo Dimmer framework adjusts GPU power limits gradually as the cluster approaches its electrical ceiling, aiming to keep workloads running rather than stop them abruptly. Rack batteries handle immediate spikes, capacitor trays smooth short peaks and facility battery systems add another layer of support.

Prometheus also brings together multiple cooling designs because its buildings and deployment timelines differ. Modular Air-Assisted Liquid Cooling serves free-air data center buildings and rapid-deployment structures. A purpose-built AI building uses Facility Liquid Cooling to support higher power density. Meta says its newest AI data centers use direct-to-chip, closed-loop liquid cooling, which circulates coolant through server cold plates and can reduce the space needed per rack. It has shared designs for its liquid-cooled IcePack network rack through the Open Compute Project.

Meta says it works with local utilities so its site pays the full cost of its energy use, and with energy developers to match consumption with new clean and renewable energy. It also uses leased capacity alongside its own facilities, and says rapid-deployment structures can bring compute online faster. These choices expand capacity, but make consistent operations across different power and cooling environments an ongoing engineering task.

What comes next

Meta plans to expand Prometheus with GB300 and Vera Rubin NVL72 systems. The Vera Rubin platform is designed to connect to the cluster’s existing network fabric. Meta says the upcoming San Miguel system will use 72 Vera Rubin GPUs, each with 288 GB of HBM4 and peak bandwidth of 19.2 TB/s per GPU.

Meta lists San Miguel at 2.52 peak exaFLOPS of FP4 training compute and 1.26 peak exaFLOPS of FP8 per rack. It also lists 216 TB/s of peak NVLink bandwidth, 1,500 TB/s of aggregate memory bandwidth and rack power above 170 kW. The system will use fully liquid-cooled compute trays. Meta estimates a 10% to 20% improvement in performance per watt over GB300 NVL72.

The deployment will test whether those per-rack gains translate into better training throughput once networking, power and cooling are included. Prometheus’s operating lessons are already informing Hyperion, Meta’s planned supercluster in Richland Parish, Louisiana. The design is expected to include closed-loop liquid cooling, updated network systems and connectivity with Entergy. Meta says it will pay the full cost of that grid connection.

For engineers evaluating the project, the clearest takeaway is that accelerator specifications alone don’t explain Prometheus. Its usable capacity depends on how effectively the system can keep thousands of GPUs synchronized, recover from faults and deliver power and cooling at scale. The next test is whether newer hardware improves that end-to-end productivity as the cluster approaches its planned size.

More on Meta’s work in New Albany

Source article | Engineering at Meta