Data centre traffic: When networks just crash

US data centre spending will hit USD 750 billion in 2026, driven by AI infrastructure demands (https://blog.apnic.net/2026/06/26/peering-capacity-at-public-internet-exchanges-what-the-data-reveals/). Forget general bandwidth optimization. The industry is now scrambling to survive the specific physical constraints of Remote Direct Memory Access (RDMA) workloads.

When RoCEv2 protocols face packet loss, they don't degrade gracefully; they fail catastrophically. This reality forces a radical departure from standard Ethernet designs. We must rethink switching layers to accommodate sustained high-volume data flows without jitter.

Dave Schaeffer of Cogent calls this expenditure a "forced activity" for digital behemoths fearing market irrelevance. With multi-year build costs projected at USD 7 trillion, getting the underlying network physics wrong is an existential threat. This isn't an upgrade cycle. It is a fundamental reconstruction of how we handle GPU communication at scale.

The Critical Role of Loss-Intolerant Traffic in AI Network Architecture

Defining Loss-Intolerant Traffic and RoCEv2 Protocols

Data flows crash instantly when packets vanish. This is the definition of loss-intolerant traffic. AI clusters relying on Remote Direct Memory Access (RDMA) create this relentless stream specifically to sidestep CPU overhead. Traditional TCP protocols absorb jitter through error correction, but GPU training workloads collapse under such latency penalties.

Operators now deploy RoCEv2 over UDP, enabling direct memory transfers between servers without kernel interference. This protocol demands a lossless network because it possesses no internal tools to repair dropped packets or fix reordering mistakes. One missing frame halts the entire distributed training process across thousands of accelerators. Industry analysis shows 2026 expenditure centers on new data centres built for intense GPU deployments. Network design must now prioritize absolute packet delivery above bandwidth efficiency.

AI Data Centre Design Shifts for GPU Cluster Performance

GPU clusters require lossless networks since RoCEv2 traffic stops dead on packet loss. This protocol offers zero retransmission logic, forcing physical infrastructure to guarantee every single delivery. Market projections suggest expenditure for 2026 will approach USD 750 billion, nearly doubling the USD 450 billion spent the previous year.

These demands challenge conventional data centre design parameters around bandwidth density, electrical reach, and optical capacity.

• Optical capacity hits walls as environments demand scaling beyond incremental upgrades.
• Power delivery strains as data centres consume more electricity than all energy-intensive manufacturing combined by the end of the 2020s.

Catastrophic Packet Loss Risks in Lossy TCP Networks

Loss-intolerant traffic fails immediately upon packet drop because RoCEv2 lacks retransmission logic. Unlike TCP, which absorbs jitter through error correction, this protocol stalls entire GPU clusters when frames vanish. The network simply cannot lose a single packet without halting distributed training processes.

Hyperscaler networks demonstrate that even minor packet reordering destroys throughput efficiency in large-scale deployments. Power constraints further limit the ability to simply oversubscribe links to mask these deficiencies. You face a binary choice: maintain familiar TCP-based tools or adopt the strict lossless architectures required for survival. Ignoring this shift renders expensive GPU hardware idle during transmission errors.

Optical Physics and Power Constraints Driving Next-Generation Data Centre Design

Thermal Limits Driving 400G Per Lane and Indium Phosphide Adoption

As AI workloads demand higher bandwidth density, the resulting thermal load requires single-mode optics capable of supporting higher speeds over 500-meter spans. We expect 400G per lane deployment in the near future. Investment in these advanced data centre optical systems is projected to grow significantly to meet this physical need.

This transition introduces complexity in fibre plant management, as single-mode requirements differ significantly from existing short-reach deployments. The physical layer ultimately constrains logical AI expansion more severely than software architecture choices.

Deploying Co-Packaged Optics to Solve GPU Cluster Power Density

Integrating Co-Packaged Optics directly onto GPU switch chassis eliminates the electrical reach limitations of pluggable transceivers. This architectural shift moves the optical engine closer to the ASIC, drastically reducing power consumption per bit compared to traditional faceplate designs.

1. Replace copper-dependent electrical interfaces with optics to extend reach beyond short interconnects.
2. Use external laser sources to mitigate heat generation within the high-density GPU rack environment.
3. Adopt lossless RoCEv2 transport protocols that demand zero packet loss across the expanded optical fabric.

The reliance on Indium Phosphide semiconductors enables the single-mode performance required for these dense deployments, whereas older Gallium Arsenide systems struggle at higher lane speeds. Operators must balance upfront capital efficiency against long-term operational flexibility. With US data centres projected to consume a significant portion of national electricity, reducing photonics power draw is not merely an optimization but a prerequisite for grid compatibility. The move to co-packaged architectures fundamentally alters the failure domain, shifting risk from individual components to integrated subsystems.

Failure Modes in High-Density Optics Without Thermal Management

High-density GPU clusters generate intense heat that degrades optical signal integrity if cooling systems fail to match power density. The inability to resolve these thermal issues leads to hardware degradation, forcing expensive replacements rather than simple reboots.

1. Monitor thermal conditions closely to prevent exceeding operating thresholds.
2. Use single-mode optics designed for dense rack layouts.
3. Design airflow paths that isolate hot exhaust from intake zones to prevent recirculation.

The economic stakes are massive, with 2026 expenditure centred on new data centres built for intense GPU deployments. This vulnerability highlights the conflict between maximizing faceplate density and maintaining sufficient thermal headroom for stable operation. Ensuring infrastructure supports the relentless demands of modern AI without succumbing to heat-induced failures remains a primary engineering challenge.

Strategic Implementation of Optical Circuit Switching and Co-Packaged Architectures

Optical Circuit Switching Mechanics for Lossless AI Fabrics

Optical Circuit Switching (OCS) establishes dedicated physical light paths between servers, effectively eliminating packet queuing and loss for RoCEv2 traffic. Electrical packet switches buffer data and risk dropping frames during GPU burst events, whereas OCS reconfigures the physical layer to create a static, collision-free tunnel. RoCEv2 protocols layered over UDP possess zero tolerance for packet reordering or jitter. Recent industry discussions highlighted that the design of data centres for Large Language Models (LLMs) now demands absolute losslessness to prevent training failures.

1. Integrate optical engines directly onto the switch substrate to overcome electrical reach limits.
2. Use dedicated light paths to support RoCEv2 streams requiring zero packet loss.
3. Address thermal dissipation limits which constrain higher capacities per fibre.

Investment in these optical systems continues to surge, reflecting strong market momentum. However, this architecture introduces a rigid dependency; upgrading the optical layer now requires replacing the entire switch unit rather than simple transceivers. Operators who frequently change vendors or throughput requirements face reduced flexibility. Bandwidth scales effectively, yet higher capacities face power density and thermal dissipation limits, creating challenges if AI algorithms shift toward different topological needs.

Longer-term multi-year build cost projections within the US are heading to USD 7 trillion, representing 2% of the US GDP for the next four years. This unusual scale creates a forced activity where digital behemoths invest primarily to avoid market extinction rather than to capture new value.

Operators implementing optical circuit switching must recognize that photonics is now a binding architectural constraint rather than a transparent transport layer. Investment in data centre optical systems continues to grow rapidly. Total per-fibre capabilities of 3.2T are foreseeable. Higher capacities face power density and thermal dissipation limits. InterLIR advises network architects to prioritize designs that acknowledge these physical power and thermal boundaries. This approach mitigates the risk of stranded assets should the current AI investment bubble correct sharply.

Investment Viability and Adoption Timelines for Emerging Optical Technologies

Defining Forced Activity AI Infrastructure Investment Drivers

Fear drives the current AI infrastructure race more than genuine market demand. Digital behemoths pour capital into proprietary platforms because staying idle invites competitors to seize their entire customer base. This defensive posture separates today's spending sprees from historical speculative bubbles built on pure exuberance. Hesitation in data centre infrastructure now carries existential weight for operators in this specific sector. Rapid expansion creates friction between immediate defensive expenditures and the need for long-term architectural stability.

AI training, inference, and agentic workloads place unusual stress on network architectures. Massive GPU clusters paired with Remote Direct Memory Access (RDMA) generate sustained, high-volume traffic that tolerates zero loss. Conventional design parameters surrounding bandwidth density, electrical reach, and optical capacity buckle under such loads. Balancing urgent GPU deployment with sustainable network design defines the modern operator's challenge. Sheer spending volume suggests immediate action, yet physical constraints of optical circuit switching demand careful phasing to prevent thermal failures. Rushing deployment often neglects how photonics now dictate architectural success more than raw compute power.

• Address limitations in silicon processors, storage systems, photonics, and switching that cannot be resolved through incremental capacity upgrades.
• Design networks to handle sustained high-volume loss-intolerant traffic generated by RDMA.
• Evaluate thermal constraints before scaling GPU density.
• Prioritize optical capacity planning over short-term compute gains.

The right choice depends on whether an operator serves immediate LLM training clusters or long-term inference workloads.

Comparing Sovereign Infrastructure Plans Against Hyperscaler Dependency

Sovereign infrastructure strategies prioritize regulatory compliance over the rapid consolidation typical of US hyperscalers. The RIPE NCC exemplifies this by exiting US-based dependencies to mitigate geopolitical risks. Funding for such initiatives relies on long-term activity plans approved by executive boards, ensuring stability rather than reactive expansion. This approach contrasts sharply with the fear-driven "forced activity" dominating current market behavior.

Operators asking when to adopt new optical technologies must weigh these divergent paths. Independent architectures sacrifice the sheer scale of hyperscale clouds for guaranteed continuity during geopolitical friction. Current internet infrastructure remains predominantly based on legacy addressing, even as the number of unallocated IPv4 addresses falls below 10%. This balanced approach secures immediate operational capacity without locking organizations into volatile external dependencies.

References

• European internet registry set to exit US hyperscalers -
• Africa's Businesses Are Online. The World Isn't Seeing Them
• APNIC Articles – APNIC: Internet addressing in the 2010s