Meta’s AI Network Spans Over 100 Km. Even Light Has Limits.

Long-distance fiber-optic cable installation connecting remote data center infrastructure, illustrating physical network latency in large AI clusters. AI Servers
Meta is connecting AI computing sites across more than 100 kilometers. But even extremely fast fiber links cannot escape the speed of light. Here's how distance changes GPU training
AI Infrastructure / The Physics of Distance

Imagine two buildings packed with AI servers. The machines are working on the same enormous model, exchanging information through optical fiber. Now move one of those buildings 100 kilometers away.

The computers have not become slower. Their GPUs are just as powerful. The fiber links can still carry an extraordinary amount of data.

Yet every message has farther to travel. Even a perfectly functioning network cannot make light arrive before the laws of physics allow it.

This is no longer merely a thought experiment. In October 2026, Meta revealed details of Prometheus, its expanding AI supercluster. Its backend aggregation network connects computing fabrics across distances ranging from 2 kilometers to more than 100 kilometers.

Meta says the first phase is complete and that Prometheus already contains hundreds of thousands of GPUs. Its longer-term goal is a system approaching one million.

The unexpected engineering problem is how to make geographically separated computers cooperate without spending too much time waiting for one another.

Key Takeaways

Four things that make distance matter

  • Meta reports links in its Prometheus aggregation network spanning from 2 km to more than 100 km.
  • Light traveling through ordinary glass fiber takes approximately five microseconds per kilometer.
  • Higher bandwidth moves more data per second, but cannot remove propagation delay.
  • Meta uses network architecture, deep buffering and communication-aware scheduling to reduce the impact of long-distance connections.

Meta is building one AI computing system across multiple sites

Traditional data centers are often described as collections of servers. At the scale of modern AI training, that description misses a crucial detail: the machines have to exchange vast amounts of information as a coordinated computing system.

A model can be distributed across multiple GPUs. Other GPUs may hold copies of model parameters while processing different training examples. During training, these groups must exchange information to keep their work consistent.

Some communication happens inside a rack, over short and extremely fast connections. Other messages travel across an entire building or between separate data centers.

Meta calls its long-distance interconnection layer Backend Aggregation, or BAG. The system links building-scale network fabrics into a larger, regional computing network.

In its October 19 engineering report, Meta described building-level Ethernet fabrics supporting up to 20,736 GPUs apiece, connected through larger aggregation switches.

BAG uses high-capacity networking equipment, including Arista chassis built around Broadcom Jericho3 switching silicon. Meta reports multi-petabit bandwidth between regional network components.

Physical Layer / Simplified Network

How a message can cross the AI cluster

  1. GPU and network interface

    A processor produces data that must be exchanged with another part of the training system.

  2. Building-scale fabric

    High-speed Ethernet switches connect GPUs within the same building or local training domain.

  3. Backend Aggregation

    Deep-buffered aggregation switches interconnect fabrics and handle traffic crossing between sites.

  4. Fiber-optic equipment transports data across the physical distance to a remote building or region.

Conceptual explanation, not Meta’s exact topology or a claim that every training message travels through all four stages.

The purpose of this architecture is not to send every GPU message over the longest available cable. It is to combine computing capacity across locations while controlling the cost of communication.

That distinction matters. A 100 km connection can be part of a large supercluster without every processor depending on that connection for every calculation.

Light travels through a vacuum at nearly 300,000 kilometers per second.

Inside conventional optical fiber, its group velocity is lower, typically around 200,000 kilometers per second. That gives a useful engineering approximation: five microseconds for each kilometer of fiber.

At first, that sounds insignificant. But compare what happens as the physical path gets longer.

il.digital / Physics Calculation

The delay of distance alone

2 km fiber ~10 µs One-way propagation
10 km fiber ~50 µs One-way propagation
100 km fiber ~500 µs One-way propagation

Illustrative calculations using 5 µs/km. These are physical propagation estimates, not measured Meta network latencies. Actual cable routes can be longer than geographic distance. Transceivers, switching, encoding and congestion add further delay.

A 100 km fiber path therefore adds approximately 0.5 milliseconds in one direction.

If a message must travel that distance and a response must return, propagation alone contributes approximately one millisecond to the round trip.

This is a lower-bound illustration for that hypothetical path, not a claim about a measured Prometheus round-trip time.

Upgrading from a fast optical link to an even faster one does not shorten the physical distance between two data centers.

The optical signal still has to travel through the glass.

Bandwidth and latency solve different problems

An 800-gigabit-per-second connection can transport an extraordinary amount of data. Yet the first bit of that data still requires time to reach the receiver.

Bandwidth describes how much information can be sent per second. Propagation latency describes how long a signal takes to cross the physical link.

They are related operationally, but increasing one does not automatically eliminate the other.

Property Question What helps?
Bandwidth How much data can pass each second? Faster optical interfaces, parallel links and network capacity.
Propagation latency How long must the signal travel through the medium? Shorter physical paths and suitable placement of communicating tasks.
Congestion What happens when several flows compete? Buffers, traffic engineering, routing and scheduling.
Reliability What happens when a link or switch fails? Multiple paths, fault detection and recovery mechanisms.

Meta uses different optical technologies for different distances. Its engineering documentation describes shorter-reach connections within and between nearby buildings, while longer links use coherent optics, including 800G-ZR technology.

Coherent optics allow powerful, high-capacity connections to extend beyond the reach of ordinary short-distance transceivers.

But even coherent optics cannot cancel the travel time imposed by distance.

Picture sending a constant stream of packets into a long optical cable. By the time the first packets reach the far end, the sender has already transmitted many more.

Engineers describe this with the bandwidth-delay product: the amount of data that can be outstanding during a chosen delay interval.

Technical Note / Illustrative Example

What 800 Gb/s means across 100 km

At 800 gigabits per second, approximately 0.5 milliseconds of one-way propagation corresponds to about 50 megabytes of data in flight.

Illustrative link rate 800 Gb/s Not a measured workload
One-way travel 0.5 ms 100 km fiber example
Data in flight ~50 MB Rate × one-way time

Decimal units. This is a bandwidth-delay calculation, not a specification of Meta’s deployed switch-buffer size. Actual buffer requirements depend on protocol behavior, traffic patterns and congestion-control design.

The calculation helps explain why long-distance congestion management becomes complicated.

If a receiver needs a sender to slow down, that instruction must travel back. Meanwhile, more data may already be moving toward the receiver.

In its 2026 OCP engineering report, Meta explained that buffer headroom has to account for cable length.

Its BAG architecture uses deep-buffered switches for long-distance traffic, while shorter building fabrics can use shallower-buffer switches designed for their local workload.

Meta also described measuring real cable length using timing probes so that buffer allocation can reflect the deployed connection rather than one universal worst-case assumption.

This is a revealing detail: even with petabit-scale network capacity, the physical length of a fiber path affects software and hardware decisions.

The slowest conversation can hold up a larger computation

Training large AI models involves more than assigning each GPU a pile of work. The machines must exchange results and synchronize at important points.

Suppose a group of GPUs finishes a computation and needs information from another group.

If the next step depends on that exchange, it cannot begin until the required communication completes.

A small delay may seem harmless once. Repeated across many synchronization phases, it can reduce the fraction of time GPUs spend doing useful arithmetic.

This is why Meta does not treat all network paths as interchangeable.

Its published design keeps particularly latency-sensitive communication close together, within tightly connected training groups.

Communication that can tolerate greater delay is more suitable for the higher aggregation layers connecting sites.

Engineering Decision / Communication Placement

Not every message belongs on the longest link

  • Inside a rack: Keep tightly coupled accelerator communication on the fastest local fabric.
  • Inside a building: Use high-bandwidth Ethernet fabrics for communication that requires frequent exchange.
  • Across sites: Route traffic through BAG when workloads and network architecture can absorb the additional latency.
  • During scheduling: Arrange work to avoid unnecessary long-distance synchronization.

Simplified description of the trade-offs discussed by Meta. Actual placement is determined by workload parallelism, available resources, network topology and fault tolerance.

In the October 2026 OCP material, Meta reported that collective communication libraries can lose substantial performance when they ignore the extra distance between network fabrics.

Its response includes routing improvements and scheduling changes that minimize how much communication must cross BAG.

The objective is not to make light faster. It is to make a training job less dependent on waiting for distant messages.

At this scale, reliability becomes part of performance

A massive training cluster contains a large number of links, optical modules, network switches and compute devices.

Even when each component is individually reliable, failures become a normal operating concern at fleet scale.

Meta reports using redundant network paths and automatic rerouting so a single fiber or port failure does not necessarily interrupt a training job.

Its team also described recovering from certain GPU failures by restarting affected training replicas while allowing other replicas to continue.

That is a different kind of optimization from simply buying a faster GPU.

A processor is useful only while it can perform productive work. Communication delays and failures can leave expensive accelerators waiting.

This connects to a different physical problem explored in our article about microscopic contamination on fiber connectors . Both stories involve optical infrastructure, but at radically different scales.

A damaged connection can affect link quality over a few centimeters. A perfectly clean connection extending across 100 km still faces unavoidable propagation delay.

AI superclusters are becoming an infrastructure problem measured in geography

For years, the most visible improvements in AI hardware came from faster processors, larger memory systems and denser racks.

Those improvements still matter. But they cannot independently solve the challenge of connecting computing capacity spread across multiple sites.

Every additional building changes the physical network. Every longer optical route affects latency. Every new network tier introduces switching, congestion-control and reliability decisions.

The same concentration of AI equipment creates challenges in other infrastructure layers. We have examined how synchronized GPU workloads affect electrical demand , and why Meta has begun liquid-cooling network switches as well as GPUs .

The network-distance problem belongs to the same physical reality.

A large AI model may run on hardware spread across an entire region. To make it behave like one useful computing system, the engineers must carefully manage which conversations take place locally and which travel farther.

Meta’s Prometheus project shows how far that design has already progressed. Its current interconnection architecture includes distances beyond 100 km, and Meta is exploring still larger geographic training systems.

The next limit on AI performance may not be how quickly a GPU can calculate. It may be how long that GPU has to wait for another one.

Sources and Methodology

  1. Meta Engineering, October 19, 2026 . Prometheus scale, BAG reach, optical interconnects, scheduling and reliability.
  2. Meta Engineering, October 12, 2026 . BAG, buffering by cable length, cross-fabric communication and multi-site networking.
  3. Meta Engineering, October 26, 2026 . OCP technical sessions on multi-site networking and optical systems.
  4. Meta Engineering, February 9, 2026 . BAG architecture, regional interconnection and deep-buffer design.
  5. Meta Engineering, October 16, 2025 . Fiber distance classes and coherent optical technology.
  6. Photonica, Fiber Latency Calculator . Optical propagation based on standard single-mode fiber group refractive index.

All Meta hardware and network details are attributed to Meta’s published engineering disclosures. Propagation delay and bandwidth-delay examples are transparent theoretical calculations, not measurements taken inside Prometheus. il.digital has not independently benchmarked Meta’s network.

Rate article
Add a comment