PCIe (peripheral component interconnect express) is an interface standard for connecting high-speed components. Every computing server or personal computer motherboard has a number of PCIe slots that can be used to add GPUs, among other add-on cards.
PCIe slots come in different physical configurations, such as: x1, x4, x8, x16. The number after the “x” tells you how many lanes that PCIe slot has. For instance, a x16 slot has 16 data-transmission lanes, connected serially with a two pair of wires, one for receiving and one to transmit.
PCIe lanes are crucial to leverage the full potential of Graphic Processing Units (GPU). In this case, we will work with an example of a computing server configured with 4 GPUs.
Each GPU needs 16x PCI-Express lanes to work at maximum capacity. with 4 GPUs the server would require a minimum of 64 PCI-E lanes available on both, the server motherboard PCI-E slots and the CPUs.
For instance, a single CPU of the series Intel Xeon E5-2600 v3 supports up to 40 PCI-E lanes, not meeting the minimum requirement of 64 PCI-E lanes to run 4 GPUs concurrently. Therefore, the server needs to be deployed with a dual (2x) CPU configuration to support the required 64 PCI-e lanes. In our Dedicated Server solutions, we are offering every multi-GPU Dedicated Servers deployed in enterprise-grade bare-metal servers with dual (2x) Intel Xeon E5-2600v3 series CPUs with a total of 80 PCI-E lanes available to the GPUs. This is the only configuration that guarantees the full performance of 4 GPUs working simultaneously.
Additionally, aside from the CPU specifications, to ensure 4 GPUs to be fully utilized, the server motherboard requires 4x PCI slots with x16 lanes specification on each of the four slots. Any other slot configuration (x4, x8) will share the available lanes among all 4 GPUs and limit the available bandwidth for the GPUs to communicate with the rest of the system, consequently degrading performance.
Many computing servers and personal computer motherboards have a variety of PCIe slot configurations that may include one or more x16 slots. But it is important to understand how many of those slots support x16 dedicated lanes. Some motherboards may have one x16 slot with two x8 slots and even a third x4 slot. Additionally, those slots may be identical in physical appearance and characteristics, but they will not support more lanes than the stated number on the manufacturer’s specifications. That means you can physically install a GPU on a x8 slot, and the GPU may run without apparent problem, but the video card will be working at half its actual capacity regarding I/O bandwidth performance.
It is important to note that in all cases the Operating System will almost always show the 4 GPUs as connected and running if the correct drivers are installed, but there will not be any message or indication about the degraded performance if not enough PCI-E lanes are available on your server to support the 4 GPUs, unless further technical diagnosis is performed.
