One Network Switch Goes Down. Does the AI Cluster Keep Running?

Conceptual illustration of two network switches between powered AI server racks, with one switch’s status lights off and the other still active. AI Servers
A spare network path can keep servers connected without preserving the job running across them. Surviving a switch failure depends on the topology, communication software and application recovery.

NVIDIA’s DGX B300 SuperPOD Ethernet reference architecture makes a specific resilience claim: a single compute-network switch failure need not abort the job. Its design gives each GPU two independent network paths, allowing the application to continue with reduced bandwidth.

That is a property of a particular end-to-end configuration. An AI cluster with several switches does not automatically have the same protection. A failed transit switch might leave another route available; a failed switch that provides a server’s only connection can isolate that server entirely.

Even when the network restores a route, an interrupted distributed job may need its communication state rebuilt or its workers restarted. “The cluster is still running” can mean the remaining servers are reachable, the current job survived, or a service recovered after an interruption. Those are different outcomes.

The spare path must start before the failed component

In a common leaf-spine network, servers connect to access switches called leaves. Leaves connect through a layer of spine switches. NVIDIA’s network reference guide explains this structure and equal-cost multipath routing, which provides multiple paths through the fabric.

Losing one spine can leave a route through another, provided the remaining links, routing and configuration support it. There may be a disruption while the failure is detected and forwarding changes, and surviving links must carry the redistributed traffic.

A server connected through only one leaf has a different problem. If that leaf fails, extra spines elsewhere cannot repair the missing first connection. Two cables into the same switch still share that switch as a failure point.

An alternate spine does not replace a missing leaf

Server group A Connected through
Leaf A
Spine 1: failed Transit path unavailable
OR — alternative transit path
Spine 2: available Alternative transit path
Server group B Connected through
Leaf B

Both transit paths depend on Leaf A and Leaf B. In this example, losing either leaf isolates its attached server group.

Simplified single-homed leaf-spine example, not a vendor wiring diagram. The spines provide alternative routes, not consecutive hops. Arrows represent bidirectional connectivity. Routing recovery and application recovery are separate; neither is simulated here.

The B300 reference design goes further than merely adding another spine. Its compute fabric uses two independent planes, with each GPU connected through both. NVIDIA describes hardware and software support that preserves operation after a single failure and allows continued work at half the original bandwidth.

That is a manufacturer’s reference-design description, not a measurement from a cluster tested for this article. It also does not mean every workload becomes exactly twice as slow. The effect on job time depends on how much that workload needs the network.

One unreachable participant can interrupt hundreds of healthy GPUs

A distributed training job is a cooperating group, not a collection of unrelated programs. Many training configurations exchange results between GPUs during each step. A collective operation is a coordinated exchange involving the participating processes.

For example, NCCL’s AllReduce operation combines values from participating GPUs and returns the combined result to each one. NCCL, the NVIDIA Collective Communication Library, requires matching participation in collective operations.

Suppose a switch failure isolates one participant in a running job. The other GPUs may remain powered and reachable through another network, yet the intended exchange cannot complete normally through the broken connection. The job may wait for a timeout or report an error rather than simply continue with fewer GPUs.

The important boundary is the communication group. A failure affecting a small part of the physical cluster can affect every process in a job that depends on that part. Other, independent jobs may continue.

A degraded link can cause trouble before a complete outage, too. Contamination in a fiber connector illustrates why link quality deserves attention alongside switch availability. Reachability by itself says little about sustained communication performance.

A working route does not automatically repair an interrupted job

There are two recovery tasks. The network restores a usable path. The application must either preserve its communication over the disruption or recover from the error it has already received.

NVIDIA’s NCCL fault-tolerance documentation describes handling fatal network errors by aborting a communicator and creating a new one. A communicator holds the information needed for a group of GPUs to exchange data. Recovery requires appropriate application behavior; restoring connectivity alone does not guarantee that the existing communicator resumes.

The same documentation includes mechanisms for forming a new communicator without failed participants. Whether training can proceed with that smaller group depends on the surrounding application, its state and its distributed algorithm. The presence of an API does not make every training script elastic.

Another recovery path is to restart workers and reload saved progress. PyTorch’s torchrun documentation describes worker-group restarts after failures and emphasizes checkpointing. The training code must save and restore the state it needs; restarting processes does not recover unsaved work.

A checkpoint is useful only if recovery can reach it. If the failed switch also cuts access to the required storage, the restart has another dependency to resolve. A design should account for that route as well as the GPU communication path.

An inference service has a different definition of survival

Consider an illustrative serving system with two independent model replicas on separate server groups. If one becomes unreachable, a correctly configured routing and health-check system can direct new requests to the surviving replica, provided it has capacity and its dependencies remain available.

That does not promise continuity for an answer already streaming from the isolated server. The connection may break, and the client may need to retry. Kubernetes’ Service documentation describes exposing workloads through a set of endpoints; an endpoint abstraction does not itself migrate a running model’s request state.

A model split across several servers is closer to the training case. Its components cooperate to produce one answer. Losing a required participant interrupts that serving group even if other servers are healthy. Separate complete replicas can provide service-level recovery, but spare GPUs elsewhere do not automatically replace the missing state.

The failed switch’s role matters as much as its location

AI installations carry more than GPU-to-GPU traffic. Storage transfers, user requests and management operations may use separate fabrics or share physical infrastructure. A management-network failure can impair monitoring and control without being the same event as losing the compute fabric. A storage-network failure creates a different set of consequences.

The switches themselves also depend on power and cooling. Meta’s liquid-cooled network rack shows how substantial that supporting infrastructure has become. Separate data paths should not be assumed independent if they depend on the same failed power source, cooling system or control component.

The meaningful test is a workload surviving a defined failure: which switch was lost, which paths remained, whether the current job continued, and what performance or progress was sacrificed. A successful ping after rerouting answers only part of the question.

One switch can fail without ending an AI job. The protection comes from independent paths, compatible communication behavior and a recovery plan that reaches all the way to application state.

Sources

Rate article
Add a comment