Why AI traffic breaks traditional network design

Enterprise networks have spent two decades being designed around a fairly stable assumption: most traffic flows north-south, between clients and servers, with predictable patterns that oversubscribed core links can handle without anyone noticing. AI training workloads break that assumption completely, and the networks built on it tend to break along with it.

The traffic pattern in distributed training is overwhelmingly east-west, GPU to GPU, often across hundreds or thousands of nodes simultaneously, synchronising gradients at the end of every training step. This is not background traffic that can tolerate congestion. A single slow link, or a single dropped packet that triggers a retransmission, can stall an entire training job until every GPU in the job catches up. At scale, that stall is multiplied across every GPU sitting idle while it waits, which turns a small networking issue into a very large cost.

Traditional oversubscribed core-and-edge designs were built on the assumption that not everyone talks to everyone else at once. AI training inverts this. Every GPU may need to communicate with every other GPU in the job, frequently and with minimal latency variance. This is why spine-leaf topologies, originally popularised in hyperscale data centres for exactly this kind of any-to-any traffic, have become the default starting point for AI-ready network design rather than an exotic option.

The second shift is the move toward lossless Ethernet, using protocols like RoCEv2 (RDMA over Converged Ethernet) to get InfiniBand-like performance characteristics, low latency, high throughput, minimal packet loss, on Ethernet infrastructure that is easier to operate and integrate with existing environments. This is not a drop-in replacement for standard Ethernet configuration. It requires careful configuration of priority flow control, congestion management and buffer sizing to actually deliver lossless behaviour under load, and getting it wrong produces a network that looks fine on a dashboard but performs badly under real training traffic.

The third shift is sheer bandwidth. 400GbE has become a baseline for new AI-focused builds, with 800GbE increasingly the target for new deployments supporting the latest generation of GPUs, where per-GPU bandwidth requirements have grown faster than most existing network refresh cycles anticipated. At the rack scale, the newest NVIDIA platforms now ship with all-to-all interconnects measured in the hundreds of terabits per second within a single rack, a figure that would have sounded like a typo two years ago and which only reinforces how far traditional core designs are from what AI infrastructure actually needs.

The practical implication for any organisation planning new AI infrastructure, or retrofitting existing infrastructure to support AI workloads, is that the network cannot be an afterthought sized to ‘whatever the existing core supports.’ It needs to be designed around the traffic pattern the GPUs will actually generate, with topology, protocol and bandwidth decisions made together rather than inherited from a network design built for a different era of enterprise traffic.

A further shift worth planning for now rather than retrofitting later is the move toward heterogeneous inference architectures, where different stages of serving a model run on different types of processor: one pool optimised for the compute-heavy prefill stage, another optimised for the bandwidth-sensitive token generation stage. This split-stage approach is becoming a standard pattern across major inference platforms, and it has a direct networking consequence. Traffic now flows not just GPU to GPU within a training job, but between distinct pools of specialised hardware that need to communicate with extremely low latency. Networks designed only around homogeneous GPU-to-GPU traffic will need to accommodate this second pattern as inference, rather than training, becomes the dominant workload in most production environments.