Solutions for Rackmounting Extra Depth Systems

Systems having socketed GPUs, such as the Tesla V100 SXM2, or the Telsa P100 SXM2, require an additional, or extended system board, onto which the socketed GPUs are seated. In this type of system, the presence of the additional or extended board requires a system chassis having more depth than most other rackmountable systems. The challenge that comes along with this is that most data centers have server cabinets, which in their most common configurations, cannot accommodate an extra depth chassis. For this reason, extra depth cabinets are usually required for rackmounting an extra depth system.

An extra depth chassis is not always required, though. Workarounds can be implemented in order to fit an extra depth chassis into a regular depth cabinet, but an uncommon cabinet configuration would be required.

Common Extra Depth Systems

Some common extra depth systems are described in Table 1. Most of the systems described in the table are GPU systems, having socketed or PCIe Tesla GPUs.

Description Height, in rackspace units Chassis Depth
NumberSmasher 1U Tesla GPU Server
with NVLink
1U 35.2″ (894mm)
39.3″ (997mm)
with rails
NumberSmasher 1U Tesla GPU Server
(4 PCIe GPUs), dual CPU sockets
1U 35.2″ (894mm)
39.3″ (997mm)
with rails
NumberSmasher 1U Tesla GPU Server,
up to 4 Tesla V100 or P100 PCIe GPUs, single CPU socket
1U 34.5″ (877mm)
NVIDIA DGX-1 3U 34.1″ (867mm)
Octoputer 4U Tesla 8-GPU Server
with NVLink
4U 31.7″ (805mm)

Table 1. Common extra depth systems

Commonly used extra depth cabinets are described in Table 2, ranging in height from 42U to 48U. Extra depth cabinets are the easiest solution for rackmounting extra depth systems. There are workarounds, however, which can be implemented for instances where a customer already has a regular depth cabinet on-site, and would prefer to use the existing cabinet, due to scarcity of floor space, for example.

Make & Part No. Height, in rackspace units Dimensions Description
APC AR3300 42U 600mm Wide x 1200mm extra depth
APC AR3305 45U 600mm Wide x 1200mm extra depth
APC AR3307 48U 600mm Wide x 1200mm extra depth

Table 2. Common extra depth cabinets

Workarounds Solutions

Workaround solutions for mounting extra depth systems in regular depth cabinets carry require considerations and conditions.

Workaround Solution #1: Remove any vertical PDUs, and replace them with horizontal PDUs

An extra depth system would be obstructed by full height vertical, “zero U”, PDUs, as it is slid toward the back of the cabinet, preventing it from sliding fully into the cabinet. All full height vertical PDUs must be removed from the cabinet, and replaced with horizontal PDUs. Because extra depth systems usually have a secondary system board, with socketed GPUs, they are usually power-dense. The NVIDIA DGX-1 GPU-accelerated system for deep learning, for example, requires 3.5kW of power, at peak workload. Tri-phase power is recommended, whenever possible, for power-dense GPU systems. With some power-dense configurations, it will not be possible to meet peak power requirements with single phase power. Along with their unusually high power density, extra depth systems will require high airflow. Using the NVIDIA DGX-1 again as an example, each of four chassis fans will each produce a maximum of 340 CFM of air flow, for a total of 1,360 CFM, per DGX-1 system. For groups of systems having high air flow requirements, the cabinet doors must be perforated.

Selecting a tri-phase horizontal PDU can be a challenge, since they do not present as many outlets as vertically mounted PDUs, and will sometimes present an outlet type which is not compatible with the inlet type on the system(s). If the entire cabinet will not be needed for mounting extra depth systems, then using a half height PDU, possibly in addition to a horizontal PDU, may be a good choice.

APC currently offers only one tri-phase horizontal PDU. Geist offers a wider variety of tri-phase horizontal PDU types. These can be searched using the Geist PDU finder. Geist does not offer a tri-phase PDU, for use with a 208V, 20Amp source. However, it offers a variety of horizontal, tri-phase PDUs, for lines carrying 30Amps, or more. Like Geist, Server Technology offers a range of tri-phase, horizontal PDUs. They offer a tri-phase PDU model which can be used with a 208V, 20Amp power source, compatible with the NEMA L21-20P plug type.

Workaround Solution #2: Replace any full height vertical PDUs with half height PDUs

Half height PDUs are typically used for shorter cabinets. But they can also be used in regular height cabinets, to allow for installation of an extra depth system chassis. If the power receptacle is below the floor, then it will be easier to mount the PDU under the extra depth systems, with the plug pointed downward. If the power receptacle is on the ceiling, then it will be easier to mount the PDU above the extra depth systems, with the plug pointed upward.

Half height vertical PDUs should only be used if the entire cabinet will not be needed for mounting extra depth systems. This is because half height vertical PDUs will still prevent installation of extra depth systems into approximately half of the cabinet’s rackspace. Some half height PDUs can be mounted on the exterior of the cabinet frame, so that they will not obstruct extra depth systems from being rackmounted.

Figure 1. Half height vertical PDU (shown horizontally)

Workaround Solution #3: Use an Extra Wide, Regular Depth Cabinet

If there is an extra wide cabinet onsite, it could possibly be used to install an extra depth system. Extra wide cabinets provide sufficient width such that vertical PDUs, mounted on the sides, will not obstruct an extra depth chassis from sliding all the way to the back of the cabinet. Removal of cabinet rear doors may still be required, however, depending on the depth of the system and cabinet. If right angle power connectors are used, then, in some cases, removal of rear doors will not be required (e.g., DGX-1 in the AR3100 cabinet). Using right angle power connectors for connecting to the PDU itself may also allow more horizontal clearance for extra depth systems. In cases where the horizontal clearance is a bit narrow, an extra depth system could be positioned vertically into another rack position, so that it will not have to squeeze between plugs connected to vertical PDUs. Positioning the system vertically to correspond with the position of a meter LCD panel on a metered PDU, for example, or between power banks, would allow for more horizontal clearance, since plugs will not be encroaching upon horizontal clearance at these vertical positions.

Make & Part No. Height, in rackspace units Dimensions Description
APC AR3150 42U 750mm Wide x 1070mm Deep extra wide
APC AR3350 42U 750mm Wide x 1200mm extra wide, extra depth

Table 3. Some extra wide cabinets

Some PDU types are deeper than others, meaning the plugs will encroach further into the horizontal clearance, since the outlets on the PDU will be at a greater distance from the side of the cabinet. Some Raritan PDUs, for example, have more depth than some APC PDUs.

Workaround Solution #4: Mount Systems at Height Corresponding to Space between PDU Power Banks

As mentioned with workaround #3, it maybe possible to mount an extra depth system at a height so that it will not run into plugs connected to PDUs. This is possible only if the PDUs are sufficiently shallow and if the system is mounted at a height corresponding to the space between PDU power banks, where no plugs protrude.

For example, an IBM Power9 system (33.3″ depth) will still fit into an APC 3100 regular depth cabinet, with four vertical AP7541 PDUs installed at the back of the cabinet, as long as it is installed at a height between the PDU power banks.

John Murphy

About John Murphy

My background in HPC includes building two clusters at the University of Massachusetts, along with doing computational research in quantum chemistry on the facilities of the Massachusetts Green High Performance Computing Center. My personal interests and academic background encompass a range of topics across science and engineering. In recent work, I used the GAMESS quantum chemistry package in order to study theoretical highly strained hydrocarbon structures, derived from prismane building blocks. I also recently authored a small software application in Python for generating amorphous cellulose within a periodic space. This application was used for generating structures for further study in NAMD and LAMMPS. Prior to doing research in Quantum and Materials Chemistry, I worked on problems related to protein folding and docking. It is very exciting, especially, to be involved with applications of GPU computing to these, as well as to other scientific questions. For several years, while not doing research, I was a consulting software engineer and built a variety of internet and desktop software applications. As an HPC Sales Specialist at Microway, I greatly look forward to advising Microway's clients in order to provide them with well-configured, optimal HPC solutions.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *