Performance Characteristics of Common Transports and Buses

Edit

<h2>Memory</h2>
The following values are measured per CPU socket. They must be doubled or quadrupled to calculate the total memory bandwidth of a multiprocessor workstation or server. For dual-processor systems, multiply by two. For quad-processor systems, multiply by four.
<table>
  <thead>
    <tr>
      <th>Type</th>
      <th># Channels</th>
      <th>Theoretical Bandwidth (unidirectional)</th>
      <th>Typical Bandwidth<br />(in Practice)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>DDR4 3200MHz</td>
      <td>Eight-Channel</td>
      <td>204.8 GB/s</td>
      <td>171.5 GB/s</td>
    </tr>
    <tr>
      <td>DDR4 2933MHz</td>
      <td>Six-Channel</td>
      <td>140.8 GB/s</td>
      <td>98 GB/s</td>
    </tr>
    <tr>
      <td>DDR4 2666MHz</td>
      <td>Six-Channel</td>
      <td>128 GB/s</td>
      <td>90 GB/s</td>
    </tr>
    <tr>
      <td>DDR4 2400MHz</td>
      <td>Quad-Channel</td>
      <td>76.8 GB/s</td>
      <td>64 GB/s</td>
    </tr>
    <tr>
      <td>DDR4 2133MHz</td>
      <td>Quad-Channel</td>
      <td>68.2 GB/s</td>
      <td>55.5 GB/s</td>
    </tr>
    <tr>
      <td>DDR3 1866MHz</td>
      <td>Quad-Channel</td>
      <td>59.7 GB/s</td>
      <td>42.8 GB/s</td>
    </tr>
    <tr>
      <td>DDR3 1600MHz</td>
      <td>Quad-Channel</td>
      <td>51.2 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1333MHz</td>
      <td>Quad-Channel</td>
      <td>42.7 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1066MHz</td>
      <td>Quad-Channel</td>
      <td>34.1 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1333MHz</td>
      <td>Triple-Channel</td>
      <td>32.0 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1066MHz</td>
      <td>Triple-Channel</td>
      <td>25.6 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 800MHz</td>
      <td>Triple-Channel</td>
      <td>19.2 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1866MHz</td>
      <td>Dual-Channel</td>
      <td>29.9 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1600MHz</td>
      <td>Dual-Channel</td>
      <td>25.6 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1333MHz</td>
      <td>Dual-Channel</td>
      <td>21.3 GB/s</td>
      <td>--</td>
    </tr>
    <tr>
      <td>DDR3 1066MHz</td>
      <td>Dual-Channel</td>
      <td>17.0 GB/s</td>
      <td>--</td>
    </tr>
  </tbody>
</table>
Theoretical memory bandwidths are calculated with: 64 bits/transfer * DDR transfers/s * number of memory channels

<hr />

<h2>PCI-Express</h2>
<table>
  <thead>
    <tr>
      <th>PCI-E Generation</th>
      <th>Lanes</th>
      <th>Theoretical Bandwidth (unidirectional)</th>
      <th>Typical Bandwidth<br />(in Practice)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Gen 1</td>
      <td>x4</td>
      <td>1,000 MB/s</td>
      <td>880 MB/s</td>
    </tr>
    <tr>
      <td>Gen 1</td>
      <td>x8</td>
      <td>2,000 MB/s</td>
      <td>1,760 MB/s</td>
    </tr>
    <tr>
      <td>Gen 1</td>
      <td>x16</td>
      <td>4,000 MB/s</td>
      <td>3,520 MB/s</td>
    </tr>
    <tr>
      <td>Gen 2</td>
      <td>x4</td>
      <td>2,000 MB/s</td>
      <td>1,600 MB/s</td>
    </tr>
    <tr>
      <td>Gen 2</td>
      <td>x8</td>
      <td>4,000 MB/s</td>
      <td>3,200 MB/s</td>
    </tr>
    <tr>
      <td>Gen 2</td>
      <td>x16</td>
      <td>8,000 MB/s</td>
      <td>6,400 MB/s</td>
    </tr>
    <tr>
      <td>Gen 3</td>
      <td>x4</td>
      <td>4,000 MB/s</td>
      <td>2,800 MB/s</td>
    </tr>
    <tr>
      <td>Gen 3</td>
      <td>x8</td>
      <td>8,000 MB/s</td>
      <td>5,600 MB/s</td>
    </tr>
    <tr>
      <td>Gen 3</td>
      <td>x16</td>
      <td>16,000 MB/s</td>
      <td>12,100 MB/s</td>
    </tr>
    <tr>
      <td>Gen 4</td>
      <td>x16</td>
      <td>32,000 MB/s</td>
      <td>26,200 MB/s</td>
    </tr>
  </tbody>
</table>

<hr />

<h2>NVIDIA GPU NVLink</h2>
The NVLink connectivity on a GPU can be split different ways depending upon the system platform design. Most NVLink 1.0 configurations split the connectivity two ways or four ways (20GB/s on each of four links). NVLink 2.0 configurations can split connectivity two, three, or six ways (25GB/s on each of six links). <a href="https://www.microway.com/preconfiguredsystems/nvidia-dgx-a100/" rel="noopener noreferrer" target="_blank">NVLink 3.0</a> supports up to twelve links (25GB/s per link).
<table>
  <thead>
    <tr>
      <th>NVLink Generation</th>
      <th>Theoretical Bandwidth* (unidirectional)</th>
      <th>Typical Bandwidth<br />(in Practice)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>NVLink 1.0 (4 bricks)</td>
      <td>80 GB/s</td>
      <td>73.4 GB/s</td>
    </tr>
    <tr>
      <td>NVLink 2.0 (6 bricks)</td>
      <td>150 GB/s</td>
      <td>143.5 GB/s</td>
    </tr>
    <tr>
      <td>NVLink 3.0 (12 bricks)</td>
      <td>300 GB/s</td>
      <td>276 GB/s</td>
    </tr>
  </tbody>
</table>
<hr />

<h2>SAS and SATA</h2>
<table>
  <thead>
    <tr>
      <th>Generation</th>
      <th>Theoretical Bandwidth (unidirectional)<br />4x wide port</th>
      <th>Typical Bandwidth (in Practice)<br />SAS / SATA</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1.5Gbps (SAS/SATA I)</td>
      <td>600 MB/s</td>
      <td>520 / 450 MB/s</td>
    </tr>
    <tr>
      <td>3Gbps (SAS/SATA II)</td>
      <td>1,200 MB/s</td>
      <td>1,140 / 990 MB/s</td>
    </tr>
    <tr>
      <td>6Gbps (SAS II/SATA III)</td>
      <td>2,400 MB/s</td>
      <td>2,280 / 1,975 MB/s</td>
    </tr>
    <tr>
      <td>12Gbps SAS</td>
      <td>4,800 MB/s</td>
      <td>3,107 / -- MB/s</td>
    </tr>
  </tbody>
</table>

<hr />

<h2>Hard Drives and SSDs</h2>
<table>
  <thead>
    <tr>
      <th>Drive Type</th>
      <th>Random IOPS</th>
      <th>Sustained Sequential I/O</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>SAS/SATA 7,200RPM</td>
      <td>70 - 175</td>
      <td>100 - 230 MB/s</td>
    </tr>
    <tr>
      <td>SAS 10,000RPM</td>
      <td>275 - 300</td>
      <td>125 - 200 MB/s</td>
    </tr>
    <tr>
      <td>SAS 15,000RPM</td>
      <td>350 - 450</td>
      <td>125 - 200 MB/s</td>
    </tr>
    <tr>
      <td>SAS/SATA Solid State Drives (SSD)</td>
      <td>15,000 - 100,000</td>
      <td>110 - 500 MB/s</td>
    </tr>
    <tr>
      <td>PCI-E Solid States (NVMe SSD)</td>
      <td>70,000 - 625,000</td>
      <td>1,100 - 3,200 MB/s</td>
    </tr>
  </tbody>
</table>

<hr />

<h2>Intel QuickPath Interconnect (QPI) and UltraPath Interconnect (UPI)</h2>
The values listed below describe a single QPI/UPI link on an Intel Xeon processor. There are typically two to three UPI links between CPU sockets, but this will vary by platform. Note that the Xeon product lines are segmented. Within a given processor series (e.g., Xeon Scalable "Cascade Lake-SP"), transfer speeds will vary from model to model.
<table>
  <thead>
    <tr>
      <th>Interconnect</th>
      <th>Transfer Speed</th>
      <th>Theoretical Bandwidth (unidirectional)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>QPI</td>
      <td>4.8 GT/s</td>
      <td>9.6 GB/s</td>
    </tr>
    <tr>
      <td>QPI</td>
      <td>5.6 GT/s</td>
      <td>11.2 GB/s</td>
    </tr>
    <tr>
      <td>QPI</td>
      <td>6.4 GT/s</td>
      <td>12.8 GB/s</td>
    </tr>
    <tr>
      <td>QPI</td>
      <td>7.2 GT/s</td>
      <td>14.4 GB/s</td>
    </tr>
    <tr>
      <td>QPI</td>
      <td>8.0 GT/s</td>
      <td>16.0 GB/s</td>
    </tr>
    <tr>
      <td>QPI</td>
      <td>9.6 GT/s</td>
      <td>19.2 GB/s</td>
    </tr>
    <tr>
      <td>UPI</td>
      <td>10.4 GT/s</td>
      <td>20.8 GB/s</td>
    </tr>
  </tbody>
</table>

<hr />

<h2>AMD Infinity Fabric</h2>
The values listed below describe a single Infinity Fabric link on an AMD EPYC processor. In dual-socket EPYC systems, there are typically three or four links between the CPU sockets. Within each EPYC CPU, each of the eight dies on the chip is connected to the I/O die via one Infinity Fabric link.
<table>
  <thead>
    <tr>
      <th>DDR4 Memory Speed</th>
      <th>Theoretical Bandwidth (unidirectional)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Zen2/Zen3 18GT/s</td>
      <td>72 GB/s</td>
    </tr>
    <tr>
      <td>Zen1 10.6GT/s</td>
      <td>42.667 GB/s</td>
    </tr>
  </tbody>
</table>
<em>Note that links between EPYC sockets include CRC overhead, which results in 8/9ths of the bandwidth values shown above (e.g., 37.9GB/s rather than 42.6GB/s).</em>

<hr />

<h2>AMD HyperTransport Link</h2>
The values listed below describe a single HyperTransport link on an AMD Opteron processor. In many of systems, there were dual HyperTransport links between the CPUs.
<table>
  <thead>
    <tr>
      <th>Generation</th>
      <th>Transfers</th>
      <th>Theoretical Bandwidth (unidirectional)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>3.1 (Socket G34)</td>
      <td>6.4 GT/s (16-bit)</td>
      <td>12.8 GB/s</td>
    </tr>
  </tbody>
</table>

<hr />

<h2>Fibre Channel (FC)</h2>
<table>
  <thead>
    <tr>
      <th>FC Rate</th>
      <th>Theoretical Bandwidth (unidirectional)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>2Gb</td>
      <td>200 MB/s</td>
    </tr>
    <tr>
      <td>4Gb</td>
      <td>400 MB/s</td>
    </tr>
    <tr>
      <td>8Gb</td>
      <td>800 MB/s</td>
    </tr>
    <tr>
      <td>16Gb</td>
      <td>1600 MB/s</td>
    </tr>
    <tr>
      <td>32Gb</td>
      <td>3200 MB/s</td>
    </tr>
  </tbody>
</table>

<hr />

<em>See also: <a href="https://www.microway.com/knowledge-center-articles/performance-characteristics-of-common-network-fabrics/" title="Performance Characteristics of Common Network Fabrics">Performance Characteristics of Common Network Fabrics</a></em>

Cancel