Detailed Specifications of the Intel Xeon E5-2600v3 "Haswell-EP" Processors

Home > Wikis > Detailed Specifications of the Intel Xeon E5-2600v3 “Haswell-EP” Processors

Detailed Specifications of the Intel Xeon E5-2600v3 “Haswell-EP” Processors

Edit

<em>This article provides in-depth discussion and analysis of the 22nm Xeon E5-2600v3 series processors (formerly codenamed "Haswell-EP"). "Haswell" processors replace the previous 22nm "Ivy Bridge" microarchitecture and are available for sale as of September 8, 2014. <strong>Note: these have since been superceded by <a href="https://www.microway.com/knowledge-center-articles/detailed-specifications-of-the-intel-xeon-e5-2600v4-broadwell-ep-processors/" target="_blank" rel="noopener">Xeon E5-2600v4 Broadwell-EP Processors</a>.</strong></em>

<h2>Important changes available in E5-2600v3 "Haswell-EP" include:</h2>
<ul>
	<li><strong>Up to 18 processor cores</strong> per socket (with options for 4-, 6-, 8-, 10-, 12-, 14- and 16-cores)</li>
	<li>Support for <strong>DDR4 memory</strong> speeds <strong>up to 2133MHz</strong></li>
	<li>Advanced Vector Extensions version 2.0 (<strong>AVX2 instructions</strong>):
	<ul>
		<li>allow <strong>256-bit wide operations for both integer and floating-point numbers</strong> (the older AVX instructions supported only floating-point operations)</li>
		<li>introduce Fused Multiply Add <strong>FMA3 instructions</strong>, which allow a multiply and an accumulate instruction to be completed in a single cycle (potentially doubling throughput for floating-point applications - <strong>up to 16 FLOPS per cycle</strong>)</li>
		<li>add support for additional instructions, including Gather and vector shift</li>
	</ul>
	</li>
	<li>Improved energy efficiency with <strong>Per Core P-States</strong> and independent uncore frequency control</li>
</ul>

With a product this complex, it’s very difficult to cover every aspect of the design. Here, we concentrate primarily on the performance of the processors for HPC applications.

<a name="performance"></a>
<h2>Exceptional Computational Performance</h2>
The Xeon E5-2600v3 processors introduce the highest performance available to date in a socketed CPU. For the first time, <strong>a single CPU is capable of more than half a TeraFLOPS</strong> (500 GFLOPS). This is made possible through the use of AVX2 with FMA3 instructions. The plot below compares the peak performance of these CPUs with and without FMA instructions:

The <em>colored bars</em> indicate performance using only AVX instructions; the <em>grey bars</em> indicate theoretical peak performance when using AVX with FMA. Note that only a small set of codes will be capable of issuing almost exclusively FMA instructions (e.g., LINPACK). Most applications will issue a variety of instructions, which will result in lower than peak FLOPS. <strong>Expect the achieved performance for well-parallelized &amp; optimized applications to fall between the grey and colored bars.</strong>

<a name="specifications"></a>
<h2>Intel Xeon E5-2600v3 Series Specifications</h2>
The tabs below compare the features and specifications of the new model line. Intel has divided the CPUs into several groups:
<ul>
	<li><strong>Standard</strong>: cost-effective CPUs with moderate performance</li>
	<li><strong>Advanced</strong>: CPUs offering the highest performance for most applications</li>
	<li><strong>High Core Count</strong>: ideal for well-parallelized applications; CPUs providing the highest number of processor cores (sometimes sacrificing clock frequency in favor of core count)</li>
	<li><strong>Frequency Optimized</strong>: ideal for non-parallel/single-threaded applications; CPUs with the highest clock speeds (sacrificing number of cores in order to provide the highest frequencies)</li>
</ul>

Although these processors introduce significant performance increases, technical readers will see that many of the changes are incremental: increased core counts, improved DDR memory speed, etc. However, processor clock speeds/frequencies have not seen significant improvements.

In fact, in some cases the CPU frequency has been lowered from the previous models. <strong>Processor frequency and Turbo Boost behavior have changed significantly with this release. Those metrics are discussed in further detail in the next section.</strong>

[tabby title="CPU Cores"]
<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Number_of_Cores.png" alt="Chart of Xeon E5-2600v3 Number of CPU Cores" width="634" height="510" class="aligncenter size-full wp-image-4600" />

[tabby title="Memory Speed"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_Memory_Performance.png" alt="Chart of Xeon E5-2600v3 Memory Performance" width="634" height="600" class="aligncenter size-full wp-image-4601" />
[tabby title="L3 Cache"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_L3_Cache_Size.png" alt="Chart of Xeon E5-2600v3 CPU L3 Cache Size" width="634" height="600" class="aligncenter size-full wp-image-4602" />
[tabby title="QPI"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_QPI_Performance.png" alt="Chart of Xeon E5-2600v3 QPI Performance" width="634" height="600" class="aligncenter size-full wp-image-4603" />
[tabby title="TDP"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_TDP_Wattage.png" alt="Chart of Xeon E5-2600v3 CPU Wattage (TDP)" width="634" height="600" class="aligncenter size-full wp-image-4604" />
[tabby title="Specifications Table"]
<table>
<thead>
<tr>
<th>Model</th> <th>AVX Frequency</th> <th>AVX Turbo Boost</th> <th>Core Count</th> <th>Memory Speed</th> <th>L3 Cache</th> <th>QPI Speed</th> <th>TDP (Watts)</th>
</tr>
</thead>
<tbody>
<tr> <td>E5-2699v3</td> <td class="vcenter" rowspan="2">1.90 GHz</td> <td class="vcenter" rowspan="2">3.30 GHz</td> <td>18</td> <td class="vcenter" rowspan="12">2133 MHz</td> <td>45MB</td> <td class="vcenter" rowspan="12">9.6 GT/s</td> <td>145W</td> </tr>
<tr> <td>E5-2698v3</td> <td>16</td> <td>40MB</td> <td>135W</td> </tr>
<tr> <td>E5-2697v3</td> <td>2.20 GHz</td> <td>3.30 GHz</td> <td class="vcenter" rowspan="3">14</td> <td class="vcenter" rowspan="3">35MB</td> <td>145W</td> </tr>
<tr> <td>E5-2695v3</td> <td>1.90 GHz</td> <td>3.00 GHz</td> <td class="vcenter" rowspan="2">120W</td> </tr>
<tr> <td>E5-2683v3</td> <td>1.70 GHz</td> <td>2.70 GHz</td> </tr>
<tr> <td>E5-2690v3</td> <td>2.30 GHz</td> <td>3.20 GHz</td> <td class="vcenter" rowspan="3">12</td> <td class="vcenter" rowspan="3">30MB</td> <td>135W</td> </tr>
<tr> <td>E5-2680v3</td> <td>2.10 GHz</td> <td>3.10 GHz</td> <td class="vcenter" rowspan="2">120W</td> </tr>
<tr> <td>E5-2670v3</td> <td>2.00 GHz</td> <td>2.90 GHz</td> </tr>
<tr> <td>E5-2687Wv3</td> <td>2.70 GHz</td> <td>3.50 GHz</td> <td class="vcenter" rowspan="3">10</td> <td class="vcenter" rowspan="3">25MB</td> <td>160W</td> </tr>
<tr> <td>E5-2660v3</td> <td>2.20 GHz</td> <td>3.10 GHz</td> <td class="vcenter" rowspan="2">105W</td> </tr>
<tr> <td>E5-2650v3</td> <td>2.00 GHz</td> <td>2.80 GHz</td> </tr>
<tr> <td>E5-2667v3</td> <td>2.70 GHz</td> <td>3.50 GHz</td> <td class="vcenter" rowspan="3">8</td> <td class="vcenter" rowspan="4">20MB</td> <td>135W</td> </tr>
<tr> <td>E5-2640v3</td> <td>2.20 GHz</td> <td>3.40 GHz</td> <td class="vcenter" rowspan="2">1866 MHz</td> <td class="vcenter" rowspan="2">8 GT/s</td> <td>90W</td> </tr>
<tr> <td>E5-2630v3</td> <td>2.10 GHz</td> <td>3.20 GHz</td> <td>85W</td> </tr>
<tr> <td>E5-2643v3</td> <td>2.80 GHz</td> <td>3.50 GHz</td> <td class="vcenter" rowspan="2">6</td> <td>2133 MHz</td> <td>9.6 GT/s</td> <td>135W</td> </tr>
<tr> <td>E5-2620v3</td> <td>2.10 GHz</td> <td>3.20 GHz</td> <td>1866 MHz</td> <td class="vcenter" rowspan="2">15MB</td> <td>8 GT/s</td> <td>85W</td> </tr>
<tr> <td>E5-2637v3</td> <td>3.20 GHz</td> <td>3.60 GHz</td> <td class="vcenter" rowspan="2">4</td> <td>2133 MHz</td> <td>9.6 GT/s</td> <td>135W</td> </tr>
<tr> <td>E5-2623v3</td> <td>2.70 GHz</td> <td>3.50 GHz</td> <td>1866 MHz</td> <td>10MB</td> <td>8 GT/s</td> <td>105W</td> </tr>
</tbody>
</table>
<em>HPC groups do not typically choose Intel's "Basic" and "Low Power" models - those skus are not shown.</em>

[tabbyending]

<a name="clockspeeds"></a>
<h2>Clock Speeds &amp; Turbo Boost in Xeon E5-2600v3 series "Haswell" processors</h2>
With each new processor line, Intel introduces new architecture optimizations. The design of the "Haswell" architecture acknowledges that highly-parallel/vectorized applications place the highest load on the processor cores (requiring more power and thus generating more heat). While a CPU core is executing intensive vector tasks (AVX instructions), the clock speed may be reduced to keep the processor within its power limits (TDP).

In effect, this may result in the processor running at a lower frequency than the "base" clock speed advertised for each model. For that reason, each "Haswell" processor model is assigned two "base" frequencies:
<ol>
	<li><strong>AVX mode</strong>: due to the higher power requirements of AVX instructions, clock speeds may be somewhat lower while executing AVX instructions *</li>
	<li><strong>Non-AVX mode</strong>: while not executing AVX instructions, the processor will operate at what would traditionally be considered the "stock" frequency</li>
</ol>
<em>* a CPU core will return to Non-AVX mode 1 millisecond after AVX instructions complete</em>

<h3>AVX and Non-AVX Turbo Boost</h3>
Just as in previous architectures, "Haswell" CPUs include the Turbo Boost feature which causes each processor core to operate well above the "base" clock speed during most operations. The precise clock speed increase depends upon the number &amp; intensity of tasks running on each CPU. With the "Haswell" architecture, Turbo Boost speed increases also depend upon the types of instructions (AVX vs. Non-AVX).

The two plots below show that processor clock speeds can be categorized as:
<ol>
	<li><strong>All cores</strong> on the CPU actively running <strong>Non-AVX</strong> instructions</li>
	<li><strong>All cores</strong> on the CPU actively running <strong>AVX</strong> instructions</li>
	<li>A <strong>single active core</strong> running <strong>Non-AVX</strong> instructions (all other cores on the CPU must be idle)</li>
	<li>A <strong>single active core</strong> running <strong>AVX</strong> instructions (all other cores on the CPU must be idle)</li>
</ol>

[tabby title="Clock Speeds for All-Core Operation"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_Speeds_AVX_and_Non-AVX_Instructions_All-Core.png" alt="Diagram of Xeon E5-2600v3 CPU Frequency Speeds (comparing AVX and Non-AVX Instructions) running on all CPU cores" width="634" height="600" class="aligncenter size-full wp-image-4606" />
[tabby title="Clock Speeds for Single-Core Operation"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_Speeds_AVX_and_Non-AVX_Instructions_Single-Core.png" alt="Diagram of Xeon E5-2600v3 CPU Frequency Speeds (comparing AVX and Non-AVX Instructions) running on a single core" width="634" height="600" class="aligncenter size-full wp-image-4605" />
[tabbyending]

Note that <strong>despite the clear rules stated above, each value is still a range of clock speeds</strong>. Because workloads are so diverse, Intel is unable to guarantee one specific clock speed for AVX or Non-AVX instructions. Users are guaranteed that cores will run within a specific range, but each application will have to be benchmarked to determine which frequencies a CPU will operate at.

When examining the differences between AVX and Non-AVX instructions, notice that Non-AVX instructions typically result in no more than a 100MHz to 200MHz increase in the highest clock speed. However, AVX instructions may cause clock speeds to drop by 300MHz to 400MHz if they are particularly intensive.

Recall that AVX2 introduces support for both integer and floating-point instructions, which means any compute-intensive application will be using such instructions (if it has been properly designed and compiled). <strong>HPC users should expect their processors to be running in AVX mode most of the time.</strong>

<h3>Top Clock Speeds for Specific Core Counts</h3>
When workloads leave some CPU cores idle, the Xeon E5-2600v3 processors are able to use that headroom to increase the clock speed of the cores which are performing work. Just as with other Turbo Boost scenarios, the precise speed increase will depend upon the CPU model. It will also depend upon how many CPU cores are active.

We advise users to consider how many CPU cores their application is able to saturate. The tabs below detail the peak Turbo Boost frequencies for each CPU model, sorted by the number of active cores:

[tabby title="1/2"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_01-02_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for single-core and dual-core applications" width="634" height="510" class="aligncenter size-full wp-image-4616" />
[tabby title="3"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_03_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for triple-core applications" width="634" height="510" class="aligncenter size-full wp-image-4617" />
[tabby title="4"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_04_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for quad-core applications" width="634" height="510" class="aligncenter size-full wp-image-4618" />
[tabby title="5"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_05_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 5-core applications" width="634" height="510" class="aligncenter size-full wp-image-4619" />
[tabby title="6"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_06_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 6-core applications" width="634" height="510" class="aligncenter size-full wp-image-4620" />
[tabby title="7"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_07_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 7-core applications" width="634" height="510" class="aligncenter size-full wp-image-4621" />
[tabby title="8"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_08_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 8-core applications" width="634" height="510" class="aligncenter size-full wp-image-4622" />
[tabby title="9/10"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_09-10_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 9- and 10-core applications" width="634" height="510" class="aligncenter size-full wp-image-4623" />
[tabby title="11/12"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_11-12_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 11- and 12-core applications" width="634" height="510" class="aligncenter size-full wp-image-4624" />
[tabby title="13/14"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_13-14_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 13- and 14-core applications" width="634" height="510" class="aligncenter size-full wp-image-4625" />
[tabby title="15/16"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_15-16_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 15- and 16-core applications" width="634" height="510" class="aligncenter size-full wp-image-4626" />
[tabby title="17+"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Frequency_17-18_core_jobs.png" alt="Chart of Xeon E5-2600v3 CPU Frequency for 17- and 18-core applications" width="634" height="510" class="aligncenter size-full wp-image-4627" />
[tabbyending]

All of the above plots show CPU frequencies for applications utilizing AVX instructions. The <em>colored bars</em> indicate the worst-case scenario - CPUs will run at least this fast. The <em>grey bars</em> indicate the expected clock speeds for most workloads.

<h2>Cost-Effectiveness and Power Efficiency of Xeon E5-2600v3 CPUs</h2>
The "Haswell-EP" processors have nearly the same price structure and power requirements as earlier Xeon E5-2600 products, so their cost-effectiveness and power-efficiency should be quite attractive to HPC users. Savvy readers may find the following facts useful:
<ul>
	<li>Although v3 Xeons follow the same price steps as their v2 counterparts, three High-Core-Count models were late additions. These models are higher performing and carry higher prices than previous E5-2600 models.</li>
	<li>The power requirement (TDP) for each model has increased by 5 Watts over the previous generation. This is due to integration of the Voltage Regulator Modules (VRMs) which were previously placed on the motherboard. Thus, CPU TDP increases 5W and motherboard TDP decreases 5W.</li>
	<li>The following graphs depict the cost-effectiveness and power-efficiency of <em>only the CPU itself</em>. In many cases, HPC users will find that once they've taken the full platform and cluster design into account, the cost-effectiveness of a higher core count CPU may be more beneficial than these plots demonstrate.</li>
</ul>

[tabby title="Performance vs. Price"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_Cost-Effectiveness.png" alt="Chart of Xeon E5-2600v3 Cost-Effectiveness (performance vs. price)" width="634" height="510" class="aligncenter size-full wp-image-4608" />
[tabby title="Performance vs. Power"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_Power-Efficiency.png" alt="Chart of Xeon E5-2600v3 Power-Efficiency" width="634" height="510" class="aligncenter size-full wp-image-4609" />
[tabby title="Processor Prices"]<img src="https://www.microway.com/wp-content/uploads/Xeon_E5-2600v3_CPU_Prices.png" alt="Chart of Xeon E5-2600v3 CPU Prices" width="634" height="510" class="aligncenter size-full wp-image-4607" />
[tabbyending]

<h2>Summary of features in Xeon E5-2600v3 "Haswell-EP" processors</h2>
In addition to the capabilities mentioned at the top of this article, these processors include many of the successful features from earlier Xeon designs. The list below provides a summary of relevant technology features:
<ul>
	<li><strong>Up to 18 processor cores</strong> per socket (with options for 4-, 6-, 8-, 10-, 12-, 14- and 16-cores)</li>
	<li>Support for Quad-channel ECC <strong>DDR4 memory speeds up to 2133MHz</strong></li>
	<li>Direct <strong>PCI-Express (generation 3.0) connections</strong> between each CPU and peripheral devices such as network adapters, GPUs and coprocessors (40 PCI-E lanes per socket)</li>
	<li><strong>Advanced Vector Extensions (AVX 2.0)</strong>:
	<ul>
		<li>effectively double the throughput of integer and floating-point operations with math units expanded from 128-bits to 256-bits</li>
		<li>introduce <strong>Fused Multiply Add</strong> (FMA3) instructions which allow a multiply and an accumulate instruction to be completed in a single cycle (effectively doubling the FLOPS/clock from 8 to 16 for each core of a CPU)</li>
		<li>add support for additional instructions, including Gather and vector shift</li>
		<li>F16C 16-bit Floating-Point conversion instructions accelerate data conversion between 16-bit and 32-bit floating point formats</li>
	</ul>
	<li><strong>Turbo Boost</strong> technology improves performance under peak loads by increasing processor clock speeds. With version 2.0, (introduced in "Sandy Bridge") clock speeds are boosted more frequently, to higher speeds and for longer periods of time. With "Haswell", top clock speeds depend upon the type of instructions (AVX vs. Non-AVX).</li>
	<li><strong>Dual Quick Path Interconnect (QPI) links</strong> between processor sockets improve communication speeds for multi-threaded applications</li>
	<li>Improved energy efficiency with <strong>Per Core P-States</strong> and independent uncore frequency control</li>
	<li>Intel <strong>Data Direct I/O Technology</strong> increases performance and reduces latency by allowing Intel ethernet controllers and adapters to talk directly with the processor cache</li>
	<li><strong>Advanced Encryption Standard New Instructions</strong> (AES-NI) accelerate encryption and decryption for fast, affordable data protection and security</li>
	<li>32-bit & 64-bit Intel <strong>Virtualization Technology</strong> (VT/VT-x) for  <strong>Directed I/O</strong> (VT-d) and <strong>Connectivity</strong> (VT-c) deliver faster performance for core virtualization processes and provide built-in hardware support for I/O virtualization.</li>
	<li>Intel <strong>APIC Virtualization</strong> (APICv) provides increased virtualization performance</li>
	<li><strong>Hyper-Threading</strong> technology allows two threads to "share" a processor core for improved resource usage. Although useful for some workloads, it is not recommended for HPC applications.</li>
</ul>

Cancel