Solutions for Rackmounting Extra Depth Systems

Edit

Extra Depth Systems having socketed GPUs, such as the <strong>Tesla V100 SXM2</strong>, or the <strong>Tesla P100 SXM2</strong>, require an additional, or extended system board, onto which the socketed GPUs are seated.  In this type of system, the presence of the additional or extended board requires a system chassis having more depth than most other rackmountable systems.  The challenge that comes along with this is that most data centers have server cabinets, which in their most common configurations, cannot accommodate an extra depth chassis.  For this reason, extra depth cabinets are usually required for rackmounting an extra depth system.

An extra depth chassis is not always required, though.  Workarounds can be implemented in order to fit an extra depth chassis into a regular depth cabinet, but an uncommon cabinet configuration would be required.

<h3>Common Extra Depth Systems</h3>
Some common extra depth systems are described in <strong>Table 1</strong>.  All of the systems described in the table are GPU systems, having socketed or PCIe Tesla GPUs.
<table style="margin: 0px; padding: 0px;">
<thead style="background-color: 4384C6; margin: 0px; padding: 0px;">
<tr style="margin: 0px; padding: 0px;">
<th style="background-color: #4384C6; color: white; font-size:104%; margin: 0px; padding: 0px;">Description</th>
<th style="background-color: #4384C6; color: white; font-size:104%; margin: 0px; padding: 0px;">Height, in rackspace units</th>
<th style="background-color: #4384C6; color: white; font-size:104%; margin: 0px; padding: 0px;">Max. GPUs</th>
<th style="background-color: #4384C6; color: white; font-size:104%; margin: 0px; padding: 0px;">Chassis Depth</th>
</tr>
</thead>
<tbody style="margin: 0px; padding: 0px;">
<tr>
<td><a href="https://www.microway.com/product/numbersmasher-1u-tesla-gpu-server-with-nvlink/" rel="noopener noreferrer" target="_blank"><strong>NumberSmasher 1U Tesla GPU Server<br/>with NVLink</strong></a></td>
<td>1U</td>
<td>4 socketed Tesla GPUs</td>
<td>35.2" (894mm)<br/>
39.3" (997mm)<br/>with rails</td>
</tr>
<tr>
<td><a href="https://www.microway.com/product/numbersmasher-1u-tesla-gpu-server-4-gpus/" rel="noopener noreferrer" target="_blank"><strong>NumberSmasher 1U Tesla GPU Server<br/>(4 PCIe GPUs), dual CPU sockets</strong></a></td>
<td>1U</td>
<td>4 PCIe Tesla GPUs</td>
<td>35.2" (894mm)<br/>
39.3" (997mm)<br/>with rails</td>
</tr>
<tr>
<td><a href="https://www.microway.com/product/numbersmasher-1u-4-gpu-server-single-root-complex/" rel="noopener noreferrer" target="_blank"><strong>NumberSmasher 1U Tesla GPU Server,<br/>up to 4 Tesla V100 or P100 PCIe GPUs, single CPU socket</strong></a></td>
<td>1U</td>
<td>4 PCIe Tesla GPUs</td>
<td>34.5" (877mm)</td>
</tr>
<tr style="margin: 0px; padding: 0px;">
<td><a href="https://www.microway.com/preconfiguredsystems/nvidia-dgx-1-deep-learning-system/" rel="noopener noreferrer" target="_blank"><strong>NVIDIA DGX-1</strong></a></td>
<td>3U</td>
<td>8 socketed Tesla GPUs</td>
<td>34.1" (867mm)</td>
</tr>
<tr>
<td><a href="https://www.microway.com/product/octoputer-4u-tesla-8-gpu-server-nvlink/" rel="noopener noreferrer" target="_blank"><strong>Octoputer 4U Tesla 8-GPU Server<br/>with NVLink</strong></a></td>
<td>4U</td>
<td>8 socketed Tesla GPUs</td>
<td>31.7" (805mm)</td>
</tr>
</tbody>
</table>
<strong>Table 1.</strong> Common extra depth systems

Commonly used extra depth cabinets are described in <strong>Table 2</strong>, ranging in height from 42U to 48U. Extra depth cabinets are the easiest solution for rackmounting extra depth systems.  There are workarounds, however, which can be implemented for instances where a customer already has a regular depth cabinet on-site, and would prefer to use the existing cabinet, due possibly, to scarcity of unused floor space. 
<table>
<thead>
<tr>
<th style="background-color: #4384C6; color: white; font-size:104%;">Make & Part No.</th>
<th style="background-color: #4384C6; color: white; font-size:104%;">Height, in rackspace units</th>
<th style="background-color: #4384C6; color: white; font-size:104%;">Dimensions</th>
<th style="background-color: #4384C6; color: white; font-size:104%;">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="http://www.apc.com/shop/in/en/products/NetShelter-SX-42U-600mm-Wide-x-1200mm-Deep-Enclosure-with-Sides-Black/P-AR3300" rel="noopener noreferrer" target="_blank"><strong>APC AR3300</strong></a></td><td>42U</td><td>600mm Wide x 1200mm</td><td>extra depth</td>
</tr>
<tr>
<td><a href="http://www.apc.com/shop/th/en/products/NetShelter-SX-45U-600mm-Wide-x-1200mm-Deep-Enclosure-with-Sides-Black/P-AR3305" rel="noopener noreferrer" target="_blank"><strong>APC AR3305</strong></a></td><td>45U</td><td>600mm Wide x 1200mm</td><td>extra depth</td>
</tr>
<tr>
<td><a href="http://www.apc.com/shop/id/en/products/NetShelter-SX-48U-600mm-Wide-x-1200mm-Deep-Enclosure-with-Sides-Black/P-AR3307" rel="noopener noreferrer" target="_blank"><strong>APC AR3307</strong></a></td><td>48U</td><td>600mm Wide x 1200mm</td><td>extra depth</td>
</tr>
</tbody>
</table>
<strong>Table 2.</strong> Common extra depth cabinets

<h2>Workarounds Solutions</h2>
Workaround solutions for mounting extra depth systems into regular depth cabinets require extra considerations and conditions.

<h3>Workaround Solution #1: Remove any vertical PDUs, and replace them with horizontal PDUs</h3>
An extra depth system would be obstructed by full height vertical, "zero U", PDUs, as it is slid toward the back of the cabinet, preventing it from sliding fully into the cabinet.  <strong>All full height vertical PDUs must be removed from the cabinet, and replaced with horizontal PDUs</strong>.  Because extra depth systems usually have a secondary system board, with socketed GPUs, they are usually power-dense.  The NVIDIA DGX-1 GPU-accelerated system for deep learning, for example, requires 3.5kW of power, at peak workload.  Tri-phase power is recommended, whenever possible, for power-dense GPU systems.  With some power-dense configurations, it will not be possible to meet peak power requirements with single-phase power.

Along with their unusually high power density, <strong>extra depth systems will require high airflow</strong>.  Using the NVIDIA DGX-1 again as an example, four chassis fans will each produce a maximum of 340 CFM of air flow, for a total of 1,360 CFM.  For groups of systems having high air flow requirements, the cabinet doors must be perforated.

Selecting a <strong>tri-phase horizontal PDU</strong> can be a challenge, since they do not present as many outlets as vertically mounted PDUs, and will sometimes present an outlet type which is not compatible with the inlet type on the system(s).  If the entire cabinet will not be needed for mounting extra depth systems, then using a half height PDU, possibly in addition to a horizontal PDU, may be a good choice.

<strong>APC</strong> currently offers only one tri-phase horizontal PDU. <strong>Geist</strong> offers a wide variety of tri-phase horizontal PDU types.  These can be searched using the <a href="http://www.geistglobal.com/pdu-search" rel="noopener noreferrer" target="_blank"><strong>Geist PDU finder</strong></a>. Geist does not offer a tri-phase PDU, for use with a 208V, 20Amp source.  However, it offers a variety of horizontal, tri-phase PDUs, for lines carrying 30Amps, or more.    Like Geist, <strong>Server Technology</strong> offers a range of tri-phase, horizontal PDUs.  They offer a <a href="https://www.servertech.com/power-distribution-unit-pdu/switched-pops/2u-horizontal-2NH3" rel="noopener noreferrer" target="_blank">tri-phase PDU model</a> which can be used with a 208V, 20Amp power source, compatible with the <a href="https://www.microway.com/knowledge-center-articles/power-and-cooling-multiple-computer-installations/" rel="noopener noreferrer" target="_blank"><strong>NEMA L21-20P</strong></a> plug type.

<h3>Workaround Solution #2: Replace any full height vertical PDUs with half height PDUs</h3>
Half height PDUs are typically used for shorter cabinets.  But they can also be used in regular height cabinets, to allow for installation of one or more extra depth system chassis.  If the power receptacle is below the floor, then it will be easier to mount the PDU under the extra depth system(s), with the plug pointed downward.  If the power receptacle is on the ceiling, then it will be easier to mount the PDU above the extra depth system(s), with the plug pointed upward.

Half height vertical PDUs should only be used if the entire cabinet will not be needed for mounting extra depth systems.  This is because half height vertical PDUs will still prevent installation of extra depth systems into approximately half of the cabinet's rackspace.  Some half height PDUs can be mounted on the exterior of the cabinet frame, so that they will not obstruct extra depth systems from being rackmounted.
<img src="https://www.microway.com/wp-content/uploads/half_height_pdu.png" />
<strong>Figure 1.</strong> Half height vertical PDU (shown horizontally)

<h3>Workaround Solution #3: Use an Extra Wide, Regular Depth Cabinet</h3>
If there is an extra wide cabinet onsite, it could possibly be used to install an extra depth system.  Extra wide cabinets provide sufficient width such that vertical PDUs, mounted at the sides, will not obstruct an extra depth chassis from sliding all the way to the back of the cabinet.  Removal of cabinet rear doors may still be required, however, depending on the depth of the system and cabinet.  If right angle power connectors are used, then, in some cases, removal of rear doors will not be required (<em>e.g.</em>, DGX-1 in the AR3100 cabinet).  Using right angle power connectors for connecting to the PDU itself may also allow more horizontal clearance for extra depth systems.  In cases where the horizontal clearance is a bit narrow, an extra depth system could be positioned vertically into another rack position, so that it will not have to squeeze between plugs connected into vertical PDUs.  Positioning the system vertically to correspond with the height of a meter LCD panel (on a metered PDU), for example, or between power banks (on most PDUs), would allow for more horizontal clearance, since plugs will not be encroaching upon horizontal clearance at these vertical positions.
<table>
<thead>
<tr>
<th style="background-color: #4384C6; color: white; font-size:104%;">Make & Part No.</th>
<th style="background-color: #4384C6; color: white; font-size:104%;">Height, in rackspace units</th>
<th style="background-color: #4384C6; color: white; font-size:104%;">Dimensions</th>
<th style="background-color: #4384C6; color: white; font-size:104%;">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="http://www.apc.com/shop/id/en/products/NetShelter-SX-42U-750mm-Wide-x-1070mm-Deep-Enclosure-with-Sides-Black/P-AR3150" rel="noopener noreferrer" target="_blank"><strong>APC AR3150</strong></a></td><td>42U</td><td>750mm Wide x 1070mm Deep</td><td>extra wide</td>
</tr>
<tr>
<td><a href="http://www.apc.com/shop/th/en/products/NetShelter-SX-42U-750mm-Wide-x-1200mm-Deep-Enclosure-with-Sides-Black/P-AR3350" rel="noopener noreferrer" target="_blank"><strong>APC AR3350</strong></a></td><td>42U</td><td>750mm Wide x 1200mm</td><td>extra wide, extra depth</td>
</tr>
</tbody>
</table>
<strong>Table 3.</strong> Some Common extra wide cabinets

Some PDU types are deeper than others, meaning the plugs will likely encroach further into the horizontal clearance, since the outlets on the PDU will be at a greater distance from the side of the cabinet.  Some Raritan PDUs, for example, have more depth than some APC PDUs.

<h3>Workaround Solution #4: Mount Systems at Height Corresponding to Space between PDU Power Banks</h3>
As mentioned with workaround #3, it maybe possible to mount an extra depth system at a height so that it will not run into plugs connected to PDUs. This is possible only if the PDUs are sufficiently shallow and if the system is mounted at a height corresponding to the space between PDU power banks, where no plugs protrude.

For example, an IBM Power9 system (33.3" depth) will still fit into an APC 3100 regular depth cabinet, with four vertical AP7541 PDUs installed at the back of the cabinet, as long as it is installed at a height between the PDU power banks.

Cancel