Power Continuity – Critical Data Centres Management
Data Centre Design at Day 1
A data centre is a controlled environment within which to conduct server-based operations. The data centre can be designed in two ways:
- A single-tenant Enterprise facility serving its own organisation
- A multi-tenant Colocation facility providing Cloud-based and/or ISP-type services
Whilst growth may be easier to forecast for an Enterprise data centre, this is not always the case with Colocation facilities.
These directly compete to sell services and virtualised server-space within a competitive market place.
Within either type of data centre, efficient floor area usage is of vital importance. The data centre will have been designed as a new build site or brown field refurbishment.
This is to house a maximum number of server cabinets or racks within the data hall area.
In most data centres, a raised access floor is used to provide cable and cooling to server cabinets.
Overhead cable trunking can be used and the cabinets to be arranged in rows.
(based on typical 600cm grid-tile references). This allows for ‘Hot-aisle, Cold-aisle’ configurations.
The server cabinets will typically be 600 or 800mm deep with up to 42U in height or greater. Space within the data hall will also be taken up by part of the cooling system. Likewise in terms of Computer Room Air Conditioners (CRACs) or Computer Room Air Handlers (CRAHs). Space also must be made via cabinets as individual systems or standalone systems.
Whether it is currently a micro-data centre (with only one server cabinet) or a large data centre housing 200 or more server cabinets. Some future expansion will have been factored into the design.
Data Centre Expansion
Most data centres will have an 18 month rolling expansion plan. This can be as little as 3 months for rapidly expanding Cloud-based service providers.
The expansion plan will allow for the take up of ‘free floor space’ to house additional servers and extra associated power and cooling services. For a micro-data centre, expansion may simply mean installing additional servers in the existing server cabinet.
For a large data centre, an addition of one or more server cabinet rows could be the case.
According to the Data Centre Dynamics Global Industry Census (2011) 58% of installed racks draw 4-5kW, 28% 5-10kW and 14% over 10kW.
Assuming a 42U high cabinet using 2U high servers, most data centre cabinets will house around twenty servers.
In terms of power, expansion can therefore range from the addition of a 300W server to the installation of another server cabinet (5-1OkW of power) or another row of say eight cabinets drawing 40-80kW. During the life of the data centre, the servers will generally be replaced at the end of the manufacturer’s warranty. This is around year 3 with more powerful (Moore’s Law) and power hungry devices. Though this can sometimes be offset by energy efficiency advancements in server design, leading to fairly constant power draws from the server cabinets.
Cooling and Power Considerations
Cooling and Power are two critical sub-systems within a data centre environment whose continuous provision is required to guarantee uptime and availability.
When a server cabinet or additional cabinet rows are added to a datacentre, the increased demand for cooling can be accommodated for. Either by existing plant or the addition of extra CRACs or CRAHs and their associated sub-systems, maybe chiller units and heat exchangers.
Extra power could be provided by the existing UPS system or by adding additional UPS capacity.
However, the total amount of power to the building will be limited by the size of the builder incomer.
Electricity will be delivered to the building from a local electrical grid connection via an incomer and distributed through LV (low voltage) switchgear and power distribution panels.
Large datacentres may also have their own direct sub-station transformer and HV (high voltage) switchgear before the LV distribution system.
The electricity supplied by the incomer may be sourced from a number of sources including:
- hydro power stations
- renewables (such as large-scale wind turbines.)
The building supply could be supplemented by on-site generated power. This can be from standby generators, solar PV, small wind turbine systems and fuel cells.
Whatever the sources of generation, the total amount of electrical power available for data centre operations is a key limiter to the overall size the data centre can grow to and it is extremely rare. It is costly to upgrade the building incomer once a data centre is operational. An alternative would be to go ‘off-the-grid’ and run on local prime power generation.
Right-Sizing And Modular Systems
The 2012 Best Practice Guide for the EU Code of Conduct on Data Centres recommends a modular approach to data centre design, matching power and cooling systems to current IT demand with the provision for easy scalability as the data centre The key word here is ‘Modular’, which can be defined as a standardised, flexible approach that provides for ease of expansion and repair.
It is the ease of expansion and therefore the flexibility of a modular approach that varies in relation to the size of the installation and within the range micro to large-scale datacentre. At the rack level every component is modular by default and it is relatively easy to install a new server, router or patch panel. The degree of flexibility decreases with scale and as one moves up the power distribution system towards the building incomer.
As discussed the size of a building incomer is fixed and it is extremely rare (and expensive) for this to be changed. Connected to the incomer supply, some datacentres may have a standby power generator to provide an alternative source of electricity when the mains power supply fails.
The generating set may be a standalone device or installed with a second or third generator in an N+X configuration . The generators may be considered modular, especially if supplied within their own containers and the arrangement allows one set be taken out of service for maintenance whilst still maintaining availability.
However, due to the physical scale of the installation, the design may not be truly flexible in terms of expansion. Each additional generating set added will impact existing space, pipework systems, fuel storage and delivery, exhausting and of course, electrical connection. As with a building incomer at this end of the power distribution chain, there is little scope generally for expansion and the flexibility offered by a modular approach is generally not easily available.
The next item to consider in the power chain is the provision for an uninterruptible power supply system. This may also be arranged in an N+X configuration and installed as a centralised or decentralised system. The term centralised is used to define a UPS system supplying site-wide power protection. De-centralised refers to the use of multiple UPS installed closer to the point of load protection i.e. at the server cabinet row level or within the server cabinets themselves.
The term ‘modular’ now becomes more relevant within the power chain when used in relation to UPS systems. At this point in the power distribution system it becomes easier to adopt the modular concept proposed by the EU Code of Conduct on Data Centres. Two UPS design formats can be used to meet the recommendations:
- Standalone UPS systems
- Modular Component UPS systems
Standalone UPS Systems
The majority of UPS systems sold are standalone UPS. They may be floor standing or rack mount. Generally, connected in a parallel/redundant N+X architecture to create systems up to 6MVA. The typical individual on-line UPS ranges in size from 800kVA to as low as 1kVA. Supplied as a single cabinet. The cabinet houses all the UPS assemblies. This is the inverter, rectifier, static switch etc. Also possibly housing the battery or require connection to an external battery cabinet, battery stand, DC flywheel or fuel-cell for a source of back-up power.
Most UPS manufacturers build parallel capability into their UPS from 10kVA upwards. (with some available from 5kVA). Parallel operation provides additional resilience. When connected in an N+X arrangement, each individual UPS system is generally referred to as a UPS module and is numbered (module 1, module 2 etc). In terms of ‘right-sizing’, a UPS system can be installed on day consisting of a single UPS or multiple UPS modules in an N+X arrangement.
Each UPS module is linked together through a communications interface cable to allow them to synchronise their outputs, load share and optimise their efficiency through the use of module ‘sleep-mode’ if available. Care has to be taken with a larger N+X arrangement. Ensuring adequate fault clearance in terms of matching supply cable lengths, and appropriately sized circuit breakers and protections.
This type of scalable standalone UPS system provides a flexible, modular approach for future expansion. The ease with which extra UPS can be added to an existing system will depend on the floor space, rack space, switchgear and power distribution connections allowed for at the design stage.
Modular Component UPS Systems
Modular component UPS systems can offer several advantages within a specific modular component. Ranging from 10-50kVA in size. Multiple modules are connected in parallel within a single system cabinet to achieve a higher kVA output and/or Level of N+X redundancy. The system cabinet is generally similar in shape and size to a server cabinet to optimise the amount of floor space required. The cabinet acts as a central hub into which UPS modules are stacked and inter-connected. The cabinet may house a battery set or be connected to an external battery source. This is connected to the mains power supply and to the Loads it will support, either directly or via some form of power distribution.
Some UPS modules are self-contained and can be installed in a server rack as standalone UPS systems. Most however require the UPS manufacturer’s cabinet which houses a centralised back plane, the firmware required to ensure output synchronisation between modules and the input and output supply distribution.
Within such a system, a communications cable provides inter-module co-ordination and standardised cable lengths are used to ensure adequate fault clearance.
With most modular component UPS systems, additional modules can be installed provided there is cabinet space available. UPS module size, weight and module accessibility can be issues, especially when working at height and may require a two-man installation team. Module connection is generally made using pre-prepared standard connection cables with ‘plug-in’ or hardwired connections. As with standalone UPS systems, additional cabinets will require extra floor space, switchgear and power distribution connections.
Modular component UPS systems can offer several advantages within a specific size range for IT operations including floor space, a more easily expandable system and the ‘hot swappable’ nature of the UPS modules themselves but at a premium price over standalone UPS systems.
There are a number of factors to consider when planning a UPS installation. The prerequisite is power quality and backup power provision. Traditionally power quality has focused on the reduction or removal of mains power problems by the UPS. This is so that the Load receives a clean, stable and regulated supply when the mains power supply is present. Backup power refers to the amount of runtime available when mains power fails. Consequently generated: standby generator, batteries, flywheel or fuel cell.
When considering UPS for data centres it is important to start with the EN/IEC definition of an on-line UPS: Voltage and Frequency Independent (VFI). On-Line UPS- also referred to as on-Line double-conversion or On-Line systems. ECO-mode operation by VFI systems is now on the green IT agenda in a bid to reduce operating costs through improved energy efficiency.
Most on-Line UPS offer this additional ECO-model functionality, essentially allowing the UPS to operate as a line interactive UPS when the mains power supply is relatively stable and clean.
For standalone On-line UPS, a further consideration is whether to adopt transformer-based or transformer-less design. The essential difference between the two types is the added robustness and galvanic isolation provided by a transformer -based UPS. This leads to increased reliability and improved power quality within electrically unstable environments .
Correspondingly, the lack of a transformer also provides standalone On-line transformer-less UPS with a more compact footprint and improved operating efficiency. Modular UPS are typically based on the transformer-less format and require an additional isolation transformer when installed in severe electrical environments.
It is important to note that whilst transformer-based UPS have tended to be less efficient than transformer-less designs, advances in power electronics and transformer design have reduced the efficiency margin between the two types.
Advantages and Disadvantages of Modular Transformer-less UPS
Floor Space Flexibility
Floor space within a datacentre is often limited. Required for revenue generating server racks. Modular component UPS systems can be expanded vertically, providing there is room within the existing cabinet for additional UPS modules. Alternatively a modular component UPS system must expand horizontally with the addition of a further UPS cabinet.
This approach can save valuable space within a data centre. Advances in the design of standalone UPS systems, has also led to the introduction of smaller-footprint systems and taller cabinet designs.
What are these?
They are standalone UPS system at 80kVA. They have a comparable footprint, similarly rated modular component UPS system and in some instances, slightly less.
To provide up to 80kVA of power the modular component UPS system cabinet is sized to house up to 4x20kVA modules. The overall physical footprint is a fixed point and comparable to a standalone UPS system. The modular component UPS system can offer a more future-proof approach with right-sizing the load on day one.
However, no UPS should ever be 100% loaded.
Why? As this leaves no room for expansion and can lead to increased component stress. At 60kVA load, the standalone 80kVA UPS may provide longer-term reliability than a similarly sized 60kVA modular UPS system. Whilst the modular system can be expanded through the addition of a further 20kVA module. Both have a maximum power output of 80kVA.
The basic argument for a modular component UPS system is:
Maximum operating efficiency:
Achieved by sizing the right UPS to the load size.
Maximum operating efficiency:
Achieved when operating at 80-100% of the design capacity.
The efficiency curve of modern standalone transformer-less On-line UPS has been greatly flattened.
Traditionally the maximum operating efficiency of this type of UPS was 94-95% at around 80-100% load.
With a wider optimum efficiency load range, high efficiency values can be achieved. Sometimes down to as little as 25% loading.
In a modular component UPS system, additional UPS modules can be added to meet future expansion. A standalone UPS can also be oversized to allow for this.
Either way, the same electrical connection costs are incurred.
Load sharing within the chosen system is also important and can affect operating efficiency. As UPS modules or standalone UPS are added to a parallel configuration the load is shared between the UPS modules.
The greater the number of UPS modules the lower the overall load on each module.
This means that one 80kVA standalone UPS operating at 80% capacity, could be more efficient than 60kVA modular UPS comprising of 4x20kVA UPS modules with one module providing N+1 redundancy.
This becomes even more important in an N+2 configuration. Ensuring that no single system is loaded by more than 50%.
The Uptime Institute defines Tier-levels with respect to power redundancy and availability. Availability is defined as: MTBF / (MTBF+MTTR) in this case.
When investigating this formula, it would be reasonable to assume that transformer-less On-line UPS, have similar or matching Mean Time Between Failure (MTBF) values.
Differences in availability figures between standalone and modular component UPS system give different MTTR or Mean Time To Repair values.
To provide a true datum line for comparison, logistics should be ignored from this calculation. Also the MTTR should be taken as the time it takes to physically install either of these two scenarios:
- Replacement UPS module into an existing modular UPS cabinet
- The swap-out of a UPS assembly or complete disconnection and reconnection of a standalone UPS
Dependable on the cable location and access to terminations, a modular component UPS system may have a 25% lower MTTR under module failure conditions . Health and safety considerations also have to be considered given the weight of the UPS module (20-40kg) and the location within the rack. (above elbow height and a safe one-person lift).
Similarly, logistics and health and safety have to be considered for the replacement of a larger standalone UPS system.
The MTTR also assumes the arrival onsite of a healthy UPS module or replacement UPS assembly or complete unit. Entire system load bank testing should be considered for any replacement component, module or system. Failure to adequately load test could lead to a catastrophic failure of the entire system.
The modular component UPS system may also have a lower overall MTBF compared to a standalone UPS system. The addition of each UPS module decreases the MTBF as the component count rises. In comparison, a standalone UPS system will have a lower component count and therefore higher MTBF for the same kVA rating.
For these reasons modular UPS cabinets tend to only house up to 6-8 modules. The lower MTBF can be compounded when batteries are taken into account. As a result when the batteries are shared between UPS modules.
Ease of Service
Modular component UPS systems are slightly easier to service and repair in situ because a failed UPS module can be ‘hot-swapped’. The failure or suspect module is then returned to a service centre for investigation. To return a standalone UPS system to active service may require a board swap.
Both ‘repairs’ will rely on sound ribbon connectors, terminal connections and screw fixtures. The modular component UPS system may require a less trained technician to perform a module swap out but any system fault condition beyond this will require a UPS engineer to attend site.
Parallel / Redundant Architecture
In a modular component UPS system it is possible to build-in a level of redundancy simply by connecting an additional UPS module. This is generally limited by design to an N+1 level of redundancy.
This may not satisfy some data centres and their service level agreements to clients.
Consequently, the level of redundancy of a modular component UPS system is limited by:
- The total overload rating capacity
- The alarm monitoring capabilities of the UPS cabinet itself.
Single Points of Failure
Standalone UPS systems operate a single uninterruptible power supply with no shared components other than a communications cable. Failure of the cable or accidental disconnection is accommodated in firmware algorithms in each UPS. Ensuring the entire system continues to function and support the load.
Modular UPS may share common components, the failure of which could result in a loss of redundancy or shutdown of the entire modular component UPS system. Common components can include:
Some modular component UPS systems can use a common three-phase AC bus with a single transfer switch.
Failure of the cabinet transfer switch could lead to a catastrophic drop of output power to the connected load.
System Control Unit:
If communication/load sharing between the UPS modules is lost, all UPS modules transferring will switch.
Common Battery Set:
A common battery can be a single point of failure.
Hence, it is especially important to perform individual battery block/string inspections in a common battery set. A failing block can weaken an entire battery string and the overall runtime available from the battery set.
Standalone UPS systems in a parallel configuration can also share a battery set and the same observations apply with regard to battery testing and inspection.
Give special consideration to modular component UPS system. Especially if a wet-cell lead acid battery is in the data centre.
Capital Costs and Return on Investment
Modular component UPS systems represent less than 5% of the global UPS market.
They have a price premium of 15-20% compared to standalone UPS systems.
What is the difference? It is mainly down to scale.
Operating efficiencies of a standalone UPS system and UPS modules within a modular component UPS system are very comparable if not near identical.
This leads to very similar Total Cost of Ownership (TCO) calculations.
What is the difference?
Only the ability of the modular component UPS system to vertically scale.
As a result, releasing floor space within a datacentre environment for revenue generating server racks, rather than critical power infrastructure. This advantage is very much within a ‘sweet spot’ size range of 20-40kVA.
They can also attract a 15-20% premium compared to a standalone UPS system.
Which type of UPS system is adopted is dependent upon where in the power distribution chain the UPS installation is to be made (power plant room or datacentre) the datacentre itself and brand reputation of the UPS manufacturer.
Most UPS manufacturers offer both standalone and modular component UPS systems.
These implement the ‘modular’ approach recommended by the ‘2012 Best Practice Guide for the EU Code of Conduct on Data Centres.’
According to Frost & Sullivan the European UPS market for modular UPS is expected to grow to £260m (US$400m) by 2017.
This demonstrates a growing acceptance of the modular component UPS system.
The growing development of small-to-medium sized datacentres and their need to rapidly expand to meet demand for Cloud-type services.
In terms of resilience, both approaches can be configured to provide similar levels of availability. Modular component UPS systems have a premium price compared to standalone UPS.
However, this will fall with increased adoption and/or the introduction of a new generation of modular technology.
This Whitepaper was published by Riello UPS.
Search – Knowledge Base