Performance Characteristics of Common Network Fabrics

Articles > Performance Characteristics of Common Network Fabrics

Ethernet

Performance of ethernet networks varies widely. Factors include the switch and NIC manufacturer, firmware settings and system/software settings. Physical layer even plays a role: 10 GigE over RJ45 (10GBase-T) has a higher latency than SFP+ Direct-Attach copper.

Contact our experts to determine a configuration which meets your requirements.

Data Rate Theoretical Bandwidth (unidirectional) End-to-End Latency Technology
Gigabit Ethernet 125 MB/s 25 ~ 65 microseconds  
10G Ethernet 1.25 GB/s 1.3 microseconds (RDMA application)
4 microseconds (sockets application)
Mellanox ConnectX-3 VPI
40G Ethernet 5 GB/s 1.3 microseconds (RDMA application)
4 microseconds (sockets application)
Mellanox ConnectX-3 VPI

InfiniBand and Omni-Path Fabrics

Although these fabrics typically offer the highest throughput and lowest latency, much depends on the configuration of the fabric and the method in which your software application accesses the fabric. The figures below describe the best possible performance – contact one of our experts to learn more.

The MPI Bandwidths are measured with large messages and MVAPICH2 MPI. The end-to-end latencies are measured with small messages and presume a single switch connecting two host adapters. Each additional switch hop will add latency (see table below).

Data Rate MPI Bandwidth (unidirectional) End-to-End Latency Generation
10Gb/s SDR 1 GB/s 2.6 microseconds Mellanox InfiniHost III
20Gb/s DDR 2 GB/s 2.6 microseconds Mellanox InfiniHost III
40Gb/s QDR 4 GB/s 1.07 microseconds Mellanox ConnectX-3
40Gb/s FDR-10 5.16 GB/s 1.07 microseconds Mellanox ConnectX-3
56Gb/s FDR 6.82 GB/s 1.07 microseconds Mellanox ConnectX-3
100Gb/s EDR 12.08 GB/s 1.01 microseconds Mellanox ConnectX-4
100Gb/s Omni-Path 12.36 GB/s 1.04 microseconds Intel 100G Omni-Path

Larger fabrics require that multiple switches be connected to provide service to all nodes. In such a fabric, each additional switch hop adds a small amount of latency.

Data Rate Hop Latency Generation
40Gb/s QDR 0.10 microseconds Mellanox InfiniScale IV
56Gb/s FDR 0.20 microseconds Mellanox SwitchX-2
100Gb/s EDR 0.09 microseconds Mellanox Switch-IB
100Gb/s Omni-Path 0.10 microseconds Intel 100G Omni-Path
200Gb/s HDR <0.09 microseconds Mellanox Quantum

See also: Performance Characteristics of Common Transports and Buses

Category: Tags:

 

Comments are closed.