As artificial intelligence infrastructure scales into configurations of staggering complexity, the limiting factor has quietly shifted from raw computational power to the speed and efficiency of data movement between chips. Interconnect technology — the layered system of protocols, switches, and signal-conditioning components that carries information across servers and memory — has emerged as the decisive variable in AI data center performance. In an era when a single rack can consume nearly 100 kilowatts and GPU clusters span tens of thousands of cards, the architecture of connection has beco
Interconnect Chips Emerge as Critical Bottleneck in AI Data Center Performance
Whoever moves data more efficiently commands the high ground
Mark
So the story here is that we've been focused on chip speed, but that's not actually the constraint anymore?
Mimi
Right. When you have tens of thousands of GPUs in a cluster, the chips themselves are fast enough. What slows everything down is getting data between them. The interconnect layer—the wiring and switching infrastructure—is now the limiting factor.
Luke
But how do we know that's actually true? Is this based on measured performance data from real clusters, or is this what the industry is saying?
Mimi
The source points to the physics: as speeds increase, signal degradation becomes severe. That's measurable. But you're right to push—the claim that interconnect is "the" bottleneck would need specific benchmarks showing where time is actually spent in training runs.
Mark
What's the practical difference between PCIe and CXL? Why does it matter which one you use?
Mimi
PCIe is the standard highway. CXL solves a specific problem: memory coherency. If you want multiple devices to share the same up-to-date data, PCIe doesn't handle that efficiently. CXL does.
Luke
And CXL is ready for production at scale? Or is this still emerging?
Mimi
The source treats it as a direction data centers are moving toward, particularly for memory pooling. But it doesn't say how widely deployed it is yet.
Mark
What about NVIDIA's NVLink? Is that competing with these standards or complementary?
Mimi
It's proprietary—NVIDIA's own high-speed rail between GPUs. It's not competing with PCIe or CXL; it's a separate ecosystem for GPU-to-GPU communication within a cluster.
Luke
So if you're building an AI cluster, you're actually using multiple interconnect technologies at once—PCIe inside the server, NVLink between GPUs, Ethernet between servers?
Mimi
Exactly. It's a layered system. Each technology solves a different communication problem at a different scale.
Mark
And chiplets—that's the next thing everyone's watching?
Mimi
It's the path forward when you hit the limits of making single chips bigger. You split the work across smaller pieces and connect them with ultra-high-speed interconnects.
Luke
But that introduces new failure points and complexity. Is the industry confident it can scale that reliably?
Mimi
The source presents it as inevitable, but doesn't address the manufacturing or reliability challenges. That's a real gap.
Le Pouls
- GPU clusters are scaling faster than the communication fabric can keep pace, creating systemic bottlenecks where chips sit idle waiting for data that cannot arrive quickly enough.
- Signal degradation at high speeds — PCIe Gen6 pushing 64 gigabits per lane — forces engineers into an arms race of retimers, redrivers, and precision clock recovery just to keep transmissions reliable.
- CXL protocol is disrupting the traditional server boundary by enabling shared memory pools across multiple machines, fundamentally redrawing assumptions about how compute resources are organized.
- Chiplet architecture and advanced 3D packaging are being pursued as the next structural response, breaking monolithic chips into interconnected fragments to extend performance gains beyond the limits of conventional manufacturing.
- Interconnect bandwidth density and power consumption are hardening into the new competitive metrics — the data center operators and chip vendors who master data movement, not just data processing, are pulling ahead.
As artificial intelligence infrastructure scales into configurations of staggering complexity, the limiting factor has quietly shifted from raw computational power to the speed and efficiency of data movement between chips. Interconnect technology — the layered system of protocols, switches, and signal-conditioning components that carries information across servers and memory — has emerged as the decisive variable in AI data center performance. In an era when a single rack can consume nearly 100 kilowatts and GPU clusters span tens of thousands of cards, the architecture of connection has become as consequential as the architecture of computation itself.
The defining constraint in modern AI data centers is no longer the performance of individual chips. When GPU clusters sprawl across tens of thousands of cards and a single rack draws nearly 100 kilowatts, what determines whether a system computes efficiently or grinds to a halt is the speed at which data moves — between chips, between servers, between memory and processors. Interconnect technology has become the quiet center of gravity in AI infrastructure.
Think of a data center as a city. PCIe is the highway connecting devices within a server. CXL is a dedicated express lane to shared memory. Retimers are signal repair stations along the route. SerDes are the compression toll booths enabling high-speed transmission. Each component serves a specific role, and each one matters when terabytes must move every second.
PCIe has evolved rapidly — from 8 gigabits per lane in Gen3 to 64 in Gen6, using a modulation technique called PAM4 that encodes data in four voltage levels rather than two. The gains are real, but so is the cost: higher speeds demand far greater signal integrity. CXL builds on PCIe's foundation while solving a problem PCIe cannot — memory coherency across devices. Its sub-protocols allow CPUs and external devices to share memory directly, enabling both memory expansion within a single server and memory pooling across multiple servers.
Between servers, Ethernet remains universal, increasingly paired with RDMA technology that allows machines to read each other's memory without CPU involvement, dramatically cutting latency. NVIDIA's NVLink and NVSwitch create proprietary high-speed rails between GPUs within a cluster. As clusters scale outward, the AllReduce operation — requiring every GPU to synchronize results — frequently becomes the true bottleneck, exposing how quickly communication limits outpace computation.
The next structural response is chiplet architecture. As monolithic chip dies approach manufacturing limits, the industry is fragmenting them into smaller chiplets reassembled through advanced packaging. Open standards like UCIe aim to unify chiplet interfaces across vendors. At the packaging level, 2.5D approaches arrange chips side by side on an interposer; 3D stacking places them vertically, connected by through-silicon vias. High Bandwidth Memory provides the ultra-wide interface GPUs require at scale.
Two metrics are crystallizing as the new measures of competitive advantage: interconnect bandwidth density — how much data can move per unit of physical area — and interconnect power consumption, now a meaningful share of total data center energy. As the race among compute chips intensifies, the ability to move data efficiently is becoming the truer test of who leads in AI infrastructure.
The bottleneck in modern AI data centers is no longer the chips themselves. When a single rack consumes nearly 100 kilowatts and GPU clusters sprawl across tens of thousands of cards, what separates a functional system from a gridlocked one is the speed at which data moves between those chips, between servers, and between memory and processors. The infrastructure that enables this movement—interconnect chips—has quietly become the most consequential variable in how efficiently AI systems actually compute.
Think of a data center as a city. The interconnect layer is its transportation system. PCIe functions as the highway connecting devices within a server. CXL acts as a dedicated express lane to shared memory warehouses. Retimers are signal repair stations positioned along the route. SerDes are the toll booths that compress data into high-speed channels. Each component serves a specific purpose in the larger architecture, and each one matters when you're trying to move terabytes of information per second.
Every interconnect solution is measured against three core metrics. Bandwidth is the volume of data that can move per unit of time—essentially the number of lanes available. Latency is how long it takes data to travel from source to destination. Throughput is what actually gets delivered in real conditions. As standards like PCIe Gen5 and Gen6 push speeds to 32 gigabits per second and beyond, signal degradation becomes a serious problem. The electrical signals weaken and distort as they travel across circuit board traces and connectors. Engineers use eye diagrams to visualize signal quality—a wider opening means the receiver can more reliably identify what it's receiving. The bit error rate quantifies how often transmission fails; lower is always better. To combat this degradation, the industry has developed retimers that reshape corrupted signals, redrivers that boost weakened ones, and clock recovery circuits that keep receivers synchronized with transmitters.
PCIe and CXL are the two primary pathways inside servers. PCIe is ubiquitous—GPUs, network cards, and storage devices almost universally connect through it. The standard has evolved rapidly: Gen3 moved data at 8 gigabits per lane, Gen4 doubled that to 16, Gen5 reached 32, and Gen6 pushes to 64 using a modulation technique called PAM4 that encodes information in four voltage levels instead of two. This doubles bandwidth at the same speed, but demands far higher signal quality. CXL builds on PCIe's foundation but solves a critical problem: PCIe cannot efficiently handle memory coherency. CXL's three sub-protocols handle device management, allow devices to access CPU cache, and let CPUs directly access memory on external devices. This opens the door to memory expansion—adding capacity to a single server through external devices—and memory pooling, where multiple servers share a common pool of memory resources.
Between servers, Ethernet remains the universal standard, though it's being augmented by specialized technologies. RDMA (Remote Direct Memory Access) allows one machine to read another's memory without CPU involvement, slashing latency. RoCE runs RDMA over Ethernet and has become the mainstream choice for AI data centers. NVIDIA's proprietary NVLink and NVSwitch create a dedicated high-speed rail between GPUs within a cluster, functioning as the traffic hub. AMD's Infinity Fabric serves a similar role. When AI clusters scale, they do so in two ways: scale-up, which connects more GPUs within a single server or rack at high speed, and scale-out, which adds more servers. In distributed training, the AllReduce operation requires every GPU to aggregate results, and this collective communication often becomes the bottleneck—GPUs compute quickly, but the data transfer cannot keep pace.
The next frontier is chiplet architecture and advanced packaging. As individual chip dies approach the physical limits of current manufacturing, the industry is splitting large chips into smaller chiplets and reassembling them through advanced packaging techniques. UCIe (Universal Chiplet Interconnect Express) is an open standard attempting to provide a unified interface for chiplets from different vendors. At the packaging level, 2.5D approaches place multiple chips side by side on an interposer, while 3D stacking places them vertically and connects them with through-silicon vias. On the memory side, HBM (High Bandwidth Memory) provides the ultra-wide interface that GPUs require, while DDR remains the standard for CPU-to-memory connections.
The interconnect supply chain now spans SerDes, PHY chips, retimers, PCIe and CXL switches, Ethernet switch ASICs, network cards, data processing units, advanced packaging, and high-speed circuit boards. Two metrics are emerging as critical: interconnect bandwidth density—how much data can move per unit of area—and interconnect power consumption, which is becoming a significant portion of total data center energy use. As the performance race among compute chips deepens, the ability to move data efficiently is becoming the true measure of competitive advantage in AI infrastructure.
Citations marquantes
What determines AI compute efficiency is often no longer the chip's peak performance itself, but the speed at which data flows between chips and between servers— Industry analysis
Envie de l'histoire complète ?
Lire l'original sur finance.biggo.com ↗
Technology