Virtual Data Centre Design

Virtual Data Centre Design A blueprint for success IT has become the back bone of every business. Advances in computing have resulted in economies of scale, allowing large companies to integrate business processes and effectively establish standard transactions through out the enterprise. Against this backdrop, building a data centre to last 10 years is a formidable challenge for major corporations Commerzbank, a major German bank, is no exception. Faced with the mammoth task of creating a future-proof data centre in London, Gary Cane, Commerzbank head of M&E, decided to use 3-D simulations techniques. This allows Commerzbank to create a virtual model of the data centre and test the validity of the design assumptions from an operational and cost point of view before committing to build the facility. Most importantly, this simulation approach can provide the basis for all future operational changes within the facility.

The Thermal Problem - Why Worry?

Power on the rise

Almost 40 years ago Gordon Moore at Intel made the famous observation that gate density in microelectronics wills double every 18 months. His prediction has been proven to be well founded and although we haven’t seen an exact parallel in the heat produced because of other improvements in techniques, the trend has certainly been followed. A decade ago a rack of electronics would typically consume a few hundred watts of power which unsurprisingly converts directly into heat. Now, a typical rack of electronics may consume 10 times that with 2 -3 kW in a rack being a common occurrence with maximum heat dissipation reaching 10kW and occasionally even more.

Can air based cooling deliver?

The simple fact is that whilst historically the industry has used liquid cooling systems to deal with hot equipment in the past, whenever possible, it has reverted to using air to cool electronics not just because of the increased financial and operational overheads of installing such a system, but also because of the not unnatural and quite possibly justified psychological aversion to mixing water and electricity.

This may seem ridiculous because by comparison with water air is almost an insulator. In numerical terms the difference seems quite phenomenal. For a 1 degree temperature rise, 1 litre of water can carry away about 3.5 times the amount of heat that 1m3 of air can remove.

While the power dissipation remained low in the data centre this was not a problem, but as the rate of heat output continues to rise, it becomes increasingly more challenging to deliver the high volumes of air required to cool the equipment. Fortunately as most of these facilities are virtually unoccupied the environments can be designed for the equipment.

The result is that we can configure the system so that the room is cool where the equipment needs it and warm elsewhere. This can mean cold aisles are cold and hot aisles really are hot provided the equipment layout and configuration practices follow the necessary procedures.

In practice while only a limited number of data centres suffer from a lack overall cooling capacity, many more are facing the increasing need to cool hot spots with locally significant requirements for cool air.

Since the historical approach has been to use an open plenum (the void created by the raised floor) the ability to control the distribution of cool air to and within the room has been limited not only because airflow is complex but also because we can’t see it and therefore have limited understanding of the complex interactions that occur.

Preventing the DC reaching the end of its life prematurely. Changes in cooling requirement also present a new business dilemma. How can you be certain the multimillion pound facility you build will have the necessary flexibility to accommodate new equipment?

The fundamental backdrop it is critical to ensure the risk of thermally induced failure-which ultimately threatens transaction processing at the heart of the business - is minimised.

What Can A Virtual Model Offer? A virtual model of a data centre removes the invisible nature of air flow and allows the company to take control of the cooling in their facility. By visualizing airflow and heat transfer you can understand the relationship between cause and effect in the data centre.

This provides the ability to induce schemes to control and distribute the cool air thus eliminating hot spots. It also allows the company to induce new equipment to the facility. These simple benefits demonstrate a key need for facility IT managers to work together in addressing a common threat – that of overheating.

To create a virtual model of a data centre that can accurately predict the key performance characteristics of the facility from an airflow and temperature perspective it is necessary to understand the complete configuration of the facility. This includes the basis and details of the cooling system as well as the equipment installed and its likely distribution.

As a result a virtual model may well offer the first real view of what the data centre look and feel like in its 3 dimensional form will. Not only does this technology offer the designer, and IT and facilities engineers a technical assessment it also provides a tool that can illustrate the situation to any members of the team who are less familiar with the technical issues but still involved in any decision making process. This can include items at the initial design stage and operational changes as they are introduced.

The Initial design stage 1. Define virtual data centre (VDC) as a model the can predict the likely

performance of the thermal environmental.

2. Help with the choice and the layout of the proposed air handler units to optimise floor void pressurisation for the ventilation system.

3. Maximise the use of available space by configuring the floor tile layouts to suit any proposed equipment layout and room architecture. This avoids the creation of hotspots and thus minimise risk.

4. Ensure efficient delivery of the cool air and effective scavenging of the warm exhaust air to improve energy efficiency

5. Define a method of balancing the power dissipation from cabinets to provide adequate cooling to all equipment at all times.

6. Create a new and scientific method of communication between the IT and Facility management.

Operational stage checklist: 1. Use the VDC to place new equipment in the facility and analyse the

environmental impact prior to making any decisions.

2. Maintain an up-to-date VDC.

Meeting the Business’ Needs

Balancing IT and Facility Requirements

The facility shown, Error! Reference source not found. 1) was the initial layout for a proposed communications room.

Figure 1 Initial Proposed Layout

The proposed room is just over 24m long band approximately 8 .5m wide. The slab to slab height is not large considering that a raised floor approach is to be used. Besides, there are several structural beams that protrude downwards and reduce the room’s overall height. These structural beams are aligned with the direction of flow for return air moving from the hot exhausts from the equipment back towards the cooling units.

Figure 2 Revised Proposed Configuration This layout was considered sufficient to house about 50 cabinets together with two short rows of patch connection racks. Although shown in the original drawing, the fire suppression equipment is not likely to be housed in the room.

Importantly the cooling strategy was to use six down-flow cooing units to pressurise a floor plenum and create cold aisles in front of the equipment. Although the orthogonal rack layouts could were initially thought to be an easy access, a final examination shown the opposite. In fact, such a layout could make it difficult to separate the hot aisles from the cold aisles.

Figure 3 Power and Data Cables As if expected, the building construction itself presented some constraints for the way in which the equipment could be arranged. For example, the equipment must be placed so that the structural columns, typical in this type of relatively open space, do not limit access to the equipment cabinets or result in restricted access unsuitable from either the equipment or health and safety perspective. In view of the overall building services it was decided that the cooling units would be better placed on the wall opposite to their original location. In addition it was decided to move the patch connection racks to the other long side wall. This has the advantage of less interruption to the main equipment cabinets. In the revised proposal, (Figure 2), equipment cabinets are arranged in two rows parallel to the length of the room. Several management practices were adopted including:

The designation of specified cable routes(Figure 3) allow:

Data into and out of the facility to/from the patch/distribution cabinets;

Power supply under the raised floor without crossing the data routes.

The use of floor bungs to limit the leakage of cool air from the ventilated floor into the room. The objective here was to limit the size of the opening around the cable to a hole no more than 25mm by 25mm.

Only fitting floor grilles in the cold aisle and to allow the warm air to return to the cooling units over the top of the equipment.

Identifying Undesirable Risk

The revised proposal (Figure 4 & Figure 3) included:

35 No. floor grilles placed in a single row between the cabinets, supplying air from 6 down-flow cooling units capable of delivering up to 40kW of cooling to return air scavenged from the room via a top return and discharging 3m3/s of air vertically downwards into the void;

Floor 25% open at the rear of the patch connection racks;

45 Cabinets housing 4kW of rack mounted equipment cooled front to back and ventilated through perforated front and rear doors;

Figure 3 Rack/Cabinet Layouts

Figure 4 ACU, Floor Grille & Rack Locations

The revised layout appears to distribute the heat more uniformly along the length of the room reflecting the installation and distribution of cooling. An early concern was whether the system could provide good pressure distribution throughout the floor void and therefore be able to deliver good cooling to all the cabinets and enclosed equipment. The chief worry was the proximity of the cooling units at the left hand end of the facility as shown where the room is most narrow. Airflow modelling uses the fundamental concepts from physics of the conservation of mass momentum and energy to predict the ventilation performance of the VDC and therefore the resulting environment for personnel and equipment alike.

Because the solution depends on such fundamentals the validity of the results will be dependent largely on the quality of the model as described by the user. For example, the way in which the equipment is installed in the cabinets and how much power each dissipates.

Results of the simulation for this revised proposal showed some very interesting features:

The short throw of the air from the cooling units, which is limited by the room width, results in high stagnation pressure by the opposite wall.

The consequence is that there is a considerable flow of cool air from the floor void into the room through the cable penetrations in the floor to the patch connection racks where cooling is not needed.

Figure 5 shows that the velocity through the patch row is almost as great as the velocity through the floor tile penetrations.

Figure 5 Flow from the Floor Void to the Room (Red) Coloured by Magnitude of Upward Velocity

The picture of air flow (Figure 7) shows how these cools (blue) air simply short circuits back to the cooling units having done little or no work.

A symptom of the problem is that the cooling load on the ACU’s to the left of the room is low (they receive cool – blue air) while the ACU’s in the middle receive warmer air (yellow) and the ACU’s to the right warmer air still (orange and red)

The difficulties at this right hand end of the room result from the fact that a disproportionate amount of air leaves the floor void to the left of the room due to the additional floor openings in the floor for the patch connection racks, and the slightly asymmetric placement of the ACU’s resulting from the room geometry.

The warmer conditions at this end of the room are further exacerbated by the higher equipment density but with gaps in one of the equipment rows.

Figure 6 Airflow Paths in the Room Coloured by Temperature

In a hot aisle / cold aisle approach equipment is always more vulnerable when it is on the end of a row because being on the end introduced another path for warm air from the hot aisle to return to the cold aisle creating undesirable mixing of the hot and cold air before it is drawn into the rack mounted equipment to do its cooling work. In this case (Figure 7) shows the temperatures in a plane just in front of the equipment.

Figure 7 Temperatures in an Elevation Just Outside the Equipment Inlet

Notice:

The air is generally warmer at the ends of the rows (including next to gaps).

The air is warmer towards the right hand end of the room (away from the patch connection equipment).

Equipment in the row with gaps is hotter than equipment in the equivalent row without gaps.

In the row with gaps a number of racks experience inlet temperatures of between 25°C and 30°C which is often considered borderline whilst the row without gaps generally experiences good conditions <25°C.

An elevation though one of the gaps clearly shows (Figure 9) the way in which the warm air from the back of the equipment penetrates into the cold aisle. Cool air from the floor grille is deflected to the inlet of the continuous row.

Figure 9 Temperatures in an Elevation Across the Two Rows of Equipment Indeed, the solution highlighted several important issues. The design team discussed these and proposed some key action points.

They are as follows:

Install shelving (Figure ) in the 2 smaller gaps not required for access to reduce the ingress of warm air from the hot aisle.

Figure 10 Shelving to be Installed in Gaps

Install a vertical barrier (Figure 11) under the floor along the line of the

floor jack / pedestals one tile in front of the patch connection racks to prevent leakage through the cable holes associated with patch equipment.

The structured cable layout under the floor means that there will only be one penetration of this barrier by cables to open and re-seal when work is done on the under-floor cabling. This was deemed more practical method of reducing the leakage from the floor void than trying to seal the individual floor penetrations for each of the patch connection racks.

Review the number of floor grilles to try and ensure best coverage of the

racks / cabinets at the end of the rows.

Figure 11 Barrier to be Installed in Floor Void

The changes have produced noticeable results. Firstly the loads on the ACU’s are much more even. This is clearly characterised by the fact that the air temperatures returning to the cooling units are much more even, (Figure 12).Notice now that the highest return air temperatures do not reach 30°C but are more in the mid twenties.

Notice also that the air at the left hand end of the room is no longer about 20°C as it was in the original proposal but that now the temperature is elevated and has done some work.

Figure 12 Airflow with the Improvements in Place The inlet temperatures, Figure are also much improved with only very limited locations where temperatures exceed the normally considered good range of up to 25°C

Figure 13 Inlet Temperatures with Improvements in Place

Benefits to the Business The detailed case shows that it pays to create a 3-D virtual model of a data centre to examine factors including airflow. Indeed, it’s this procedure that increased the certainty the facility layout will work. More important, should Commerzbank choose to proceed with the construction of this facility they can be confident that the data centre will satisfy performance requirements both now and in the future as more equipment is housed there. In conclusion, undertaking this analysis has produced several business benefits. We have identified modifications that will minimise the risk of equipment overheating. This exercise also ensures that all cabinets housed within the facility are exposed to good inlet conditions, a measure that cuts the time spent deciding where to install equipment. But there are still opera issues. Clearly, it will still be important to follow god practice when determining the configuration of the rack mounted equipment within the cabinet and the total load per cabinet. Nonetheless, this detailed analysis has successfully brought the IT and Facilities Management teams together to achieve a common vision – and developed a virtually future proof data centre.