Detailed Specifications of the Intel Xeon E5-4600 v3 “Haswell-EP” Processors

Detailed Specifications of the Intel Xeon E5-4600 v3 “Haswell-EP” Processors

<em>This article provides in-depth discussion and analysis of the 22nm Xeon E5-4600 v3 series processors (formerly codenamed "Haswell-EP"). "Haswell" processors replace the previous 22nm "Ivy Bridge" microarchitecture and are available for sale as of June 1, 2015. For an introduction, read our blog post <a href="http://www.microway.com/hpc-tech-tips/intel-xeon-e5-4600v3-cpu-review/" title="Intel Xeon E5-4600v3 “Haswell” 4-socket CPU Review" target="_blank">Xeon E5-4600v3 4-socket CPU Review</a></em>

<h2>Important changes available in E5-4600 v3 "Haswell-EP" include:</h2>
<ul>
<li><strong>Up to 18 processor cores</strong> per socket (with options for 4-, 6-, 8-, 10-, 12-, 14- and 16-cores)</li>
<li>Support for <strong>DDR4 memory</strong> speeds <strong>up to 2133MHz</strong></li>
<li>Advanced Vector Extensions version 2.0 (<strong>AVX2 instructions</strong>):
<ul>
<li>allow <strong>256-bit wide operations for both integer and floating-point numbers</strong> (the older AVX instructions supported only floating-point operations)</li>
<li>introduce Fused Multiply Add <strong>FMA3 instructions</strong>, which allow a multiply and an accumulate instruction to be completed in a single cycle (potentially doubling throughput for floating-point applications – <strong>up to 16 FLOPS per cycle</strong>)</li>
<li>add support for additional instructions, including Gather and vector shift</li>
</ul>
</li>
<li>Improved energy efficiency with <strong>Per Core P-States</strong> and independent uncore frequency control</li>
</ul>

With a product this complex, it’s very difficult to cover every aspect of the design. Here, we concentrate primarily on the performance of the processors for HPC applications.

<a name="performance"></a>
<h2>Exceptional Computational Performance</h2>
The Xeon E5-4600 v3 processors provide some of the highest performance available to date in a socketed CPU (similar to their dual-socket "Haswell-EP" counterparts). For the first time, this architecture offers <strong>a single CPU capable of more than half a TeraFLOPS</strong> (500 GFLOPS) and total system performance over 2 TFLOPS!. This is made possible through the use of AVX2 with FMA3 instructions. The plot below compares the peak performance of a single CPU with and without FMA instructions:

<img src="http://www.microway.com/wp-content/uploads/Xeon_E5-4600v3_Theoretical_Peak_Performance.png" alt="Chart of Xeon E5-4600 v3 Theoretical Peak Performance in GigaFLOPS" width="634" height="600" class="aligncenter size-full wp-image-5369" />

The <em>colored bars</em> indicate performance using only AVX instructions; the <em>grey bars</em> indicate theoretical peak performance when using AVX with FMA. Note that only a small set of codes will be capable of issuing almost exclusively FMA instructions (e.g., LINPACK). Most applications will issue a variety of instructions, which will result in lower than peak FLOPS. <strong>Expect the achieved performance for well-parallelized &amp; optimized applications to fall between the grey and colored bars.</strong>

<a name="specifications"></a>
<h2>Intel Xeon E5-4600 v3 Series Specifications</h2>
The tabs below compare the features and specifications of the new model line. Intel has divided the CPUs into several groups:
<ul>
<li><strong>Standard</strong>: cost-effective CPUs with moderate performance</li>
<li><strong>Advanced</strong>: CPUs offering the highest performance for most applications</li>
<li><strong>High Core Count</strong>: ideal for well-parallelized applications; CPUs providing the highest number of processor cores (sometimes sacrificing clock frequency in favor of core count)</li>
<li><strong>Frequency Optimized</strong>: ideal for non-parallel/single-threaded applications; CPUs with the highest clock speeds (sacrificing number of cores in order to provide the highest frequencies)</li>
</ul>

Although these processors introduce significant performance increases, technical readers will see that many of the changes are incremental: increased core counts, improved DDR memory speed, etc. However, processor clock speeds/frequencies have not seen significant improvements.

In fact, in some cases the CPU frequency has been lowered from the previous models. <strong>Processor frequency and Turbo Boost behavior have changed significantly with this release. Those metrics are discussed in further detail in the next section.</strong>

<a name="clockspeeds"></a>
<h2>Clock Speeds &amp; Turbo Boost in Xeon E5-4600 v3 series "Haswell" processors</h2>
With each new processor line, Intel introduces new architecture optimizations. The design of the "Haswell" architecture acknowledges that highly-parallel/vectorized applications place the highest load on the processor cores (requiring more power and thus generating more heat). While a CPU core is executing intensive vector tasks (AVX instructions), the clock speed may be reduced to keep the processor within its power limits (TDP).

In effect, this may result in the processor running at a lower frequency than the "base" clock speed advertised for each model. For that reason, each "Haswell" processor model is assigned two "base" frequencies:
<ol>
<li><strong>AVX mode</strong>: due to the higher power requirements of AVX instructions, clock speeds may be somewhat lower while executing AVX instructions *</li>
<li><strong>Non-AVX mode</strong>: while not executing AVX instructions, the processor will operate at what would traditionally be considered the "stock" frequency</li>
</ol>
<em>* a CPU core will return to Non-AVX mode 1 millisecond after AVX instructions complete</em>

<h3>AVX and Non-AVX Turbo Boost</h3>
Just as in previous architectures, "Haswell" CPUs include the Turbo Boost feature which causes each processor core to operate well above the "base" clock speed during most operations. The precise clock speed increase depends upon the number &amp; intensity of tasks running on each CPU. With the "Haswell" architecture, Turbo Boost speed increases also depend upon the types of instructions (AVX vs. Non-AVX).

The two plots below show that processor clock speeds can be categorized as:
<ol>
<li><strong>All cores</strong> on the CPU actively running <strong>Non-AVX</strong> instructions</li>
<li><strong>All cores</strong> on the CPU actively running <strong>AVX</strong> instructions</li>
<li>A <strong>single active core</strong> running <strong>Non-AVX</strong> instructions (all other cores on the CPU must be idle)</li>
<li>A <strong>single active core</strong> running <strong>AVX</strong> instructions (all other cores on the CPU must be idle)</li>
</ol>

Note that <strong>despite the clear rules stated above, each value is still a range of clock speeds</strong>. Because workloads are so diverse, Intel is unable to guarantee one specific clock speed for AVX or Non-AVX instructions. Users are guaranteed that cores will run within a specific range, but each application will have to be benchmarked to determine which frequencies a CPU will operate at.

When examining the differences between AVX and Non-AVX instructions, notice that Non-AVX instructions do not result in dramatically higher Turbo Boost speeds. With the exception of the E5-4620 v3, none of the grey bars rises any higher than the colored bars. Thus, for most CPUs the maximum possible Turbo Boost speed is the same when using AVX and Non-AVX instructions. However, heavy usage of AVX instructions may reduce the clock speed by as much as 300MHz.

Recall that AVX2 introduces support for both integer and floating-point instructions, which means any compute-intensive application will be using such instructions (if it has been properly designed and compiled). <strong>HPC users should expect their processors to be running in AVX mode most of the time.</strong>

Of course, it is worth remembering that the usage of AVX instructions can result in as much as a 100% increase in performance. It is much better to leverage AVX instructions – gaining the 100% increase in instruction throughput and suffering the small 5% to 15% CPU clock speed penalty. It would be unwise to turn off AVX with the expectation that overall performance would increase.

<h3>Top Clock Speeds for Specific Core Counts</h3>
When workloads leave some CPU cores idle, the Xeon E5-4600 v3 processors are able to use that headroom to increase the clock speed of the cores which are performing work. Just as with other Turbo Boost scenarios, the precise speed increase will depend upon the CPU model. It will also depend upon how many CPU cores are active.

We advise users to consider how many CPU cores their application is able to saturate. The tabs below detail the peak Turbo Boost frequencies for each CPU model, sorted by the number of active cores:

All of the above plots show CPU frequencies for applications utilizing AVX instructions. The <em>colored bars</em> indicate the worst-case scenario – CPUs will run at least this fast. The <em>grey bars</em> indicate the expected clock speeds for most workloads.

<h2>Cost-Effectiveness and Power Efficiency of Xeon E5-4600 v3 CPUs</h2>
The "Haswell-EP" processors have nearly the same price structure and power requirements as earlier Xeon E5-4600 products, so their cost-effectiveness and power-efficiency should be quite attractive to HPC users. Savvy readers may find the following facts useful:
<ul>
<li>The Xeon E5-4627 v3 CPUs are typically optimized for HPC workloads. Additionally, they feature pricing attractive to HPC groups.</li>
<li>The power requirement (TDP) for each model has increased by 5 Watts over the previous generation. This is due to integration of the Voltage Regulator Modules (VRMs) which were previously placed on the motherboard. Thus, CPU TDP increases 5W and motherboard TDP decreases 5W.</li>
<li>The following graphs depict the cost-effectiveness and power-efficiency of <em>only the CPU itself</em>. In many cases, HPC users will find that once they’ve taken the full platform and cluster design into account, the cost-effectiveness of a higher core count CPU may be more beneficial than these plots demonstrate.</li>
</ul>

<h2>Summary of features in Xeon E5-4600 v3 "Haswell-EP" processors</h2>
In addition to the capabilities mentioned at the top of this article, these processors include many of the successful features from earlier Xeon designs. The list below provides a summary of relevant technology features:
<ul>
<li><strong>Up to 18 processor cores</strong> per socket (with options for 4-, 6-, 8-, 10-, 12-, 14- and 16-cores)</li>
<li>Support for Quad-channel ECC <strong>DDR4 memory speeds up to 2133MHz</strong></li>
<li>Direct <strong>PCI-Express (generation 3.0) connections</strong> between each CPU and peripheral devices such as network adapters, GPUs and coprocessors (40 PCI-E lanes per socket)</li>
<li><strong>Advanced Vector Extensions (AVX 2.0)</strong>:
<ul>
<li>effectively double the throughput of integer and floating-point operations with math units expanded from 128-bits to 256-bits</li>
<li>introduce <strong>Fused Multiply Add</strong> (FMA3) instructions which allow a multiply and an accumulate instruction to be completed in a single cycle (effectively doubling the FLOPS/clock from 8 to 16 for each core of a CPU)</li>
<li>add support for additional instructions, including Gather and vector shift</li>
<li>F16C 16-bit Floating-Point conversion instructions accelerate data conversion between 16-bit and 32-bit floating point formats</li>
</ul>
<li><strong>Turbo Boost</strong> technology improves performance under peak loads by increasing processor clock speeds. With version 2.0, (introduced in "Sandy Bridge") clock speeds are boosted more frequently, to higher speeds and for longer periods of time. With "Haswell", top clock speeds depend upon the type of instructions (AVX vs. Non-AVX).</li>
<li>Faster <strong>Quick Path Interconnect (QPI) links</strong> between processor sockets improve communication speeds for multi-threaded applications</li>
<li>Improved energy efficiency with <strong>Per Core P-States</strong> and independent uncore frequency control</li>
<li>Intel <strong>Data Direct I/O Technology</strong> increases performance and reduces latency by allowing Intel ethernet controllers and adapters to talk directly with the processor cache</li>
<li><strong>Advanced Encryption Standard New Instructions</strong> (AES-NI) accelerate encryption and decryption for fast, affordable data protection and security</li>
<li>32-bit & 64-bit Intel <strong>Virtualization Technology</strong> (VT/VT-x) for <strong>Directed I/O</strong> (VT-d) and <strong>Connectivity</strong> (VT-c) deliver faster performance for core virtualization processes and provide built-in hardware support for I/O virtualization.</li>
<li>Intel <strong>APIC Virtualization</strong> (APICv) provides increased virtualization performance</li>
<li><strong>Hyper-Threading</strong> technology allows two threads to "share" a processor core for improved resource usage. Although useful for some workloads, it is not recommended for HPC applications.</li>
</ul>