Critical Data Centres Management
Critical Data Centres need UPS modules and standby power generators to provide an emergency power source of electricity when the mains power supply fails. Business critical IT can not afford down time. Critical data centre servers handle every facet of our daily lives including personal data, email communication, financial controls, sales and purchasing, inventory, communications together with secure access. Data Centre Management includes Server monitoring 24/7 to protect against cyber attacks as well as air-conditioning to prevent overheating server racks. Businesses infrastructure is ever-changing and data centres have to meet their IT requirements. Companies don’t start off with Hundred Thousand Pound Data Centres in year 1, Data centres evolve starting from small server cabinets. When Businesses expand, their requirements overwhelm the server capacities therefore slowing future progress. For that reason Critical Data Centres are a wise investment when planning the next 5 years of growth.
Critical Data Centre Design at Day 1
A data centre is a controlled environment within which to conduct server-based operations.
Two Data Centres Designs:
- A single-tenant Enterprise facility serving its own organisation
- A multi-tenant Colocation facility providing Cloud-based and/or ISP-type services
Whilst growth may be easier to forecast for an Enterprise data centre, this is not always the case with Colocation facilities.
These directly compete to sell services and virtualised server-space within a competitive market place.
Within either type of critical data centre, efficient floor area usage is vital importance.
Also, raised floors provide undisputed cable access to server cabinets.
Alternatively, overhead cable trunking. Cabinets also to be arranged in rows.
The data centre is designed as a:
- new build site or
- brown field refurbishment
Maximising the number of server cabinets or racks within the data hall area.
(based on typical 600cm grid-tile references). This allows for ‘Hot-aisle, Cold-aisle’ configurations.
The server cabinets will typically be 600 or 800mm deep with up to 42U in height or greater. Space within the data hall will include the individual cooling system parts. Likewise in terms of Computer Room Air Conditioners (CRACs) or Computer Room Air Handlers (CRAHs). Space must be made via cabinets. Individual systems or standalone systems.
Whether it is currently a micro-data centre (with only one server cabinet) or a large data centre housing 200 or more server cabinets.
Critical Data Centre Expansion
Most critical data centres will have an 18 month rolling expansion plan. This can be as little as 3 months for rapidly expanding Cloud-based service providers.
The expansion plan will allow for the take up of ‘free floor space’ to house additional servers and extra associated power and cooling services. For a micro-data centre, expansion may simply mean installing additional servers in the existing server cabinet.
For a large data centre, an addition of one or more server cabinet rows could be the case.
According to the Data Centre Dynamics Global Industry Census (2011) 58% of installed racks draw 4-5kW, 28% 5-10kW and 14% over 10kW.
Assuming a 42U high cabinet using 2U high servers, most data centre cabinets will house around twenty servers.
In terms of power, expansion can therefore range from the addition of a 300W server to the installation of another server cabinet (5-1OkW of power) or another row of say eight cabinets drawing 40-80kW. During the life of the data centre, the servers will generally be replaced at the end of the manufacturer’s warranty. This is around year 3 with more powerful (Moore’s Law) and power hungry devices. Though this can sometimes be offset by energy efficiency advancements in server design, leading to fairly constant power draws from the server cabinets.
Cooling and Power Considerations
Cooling and Power are two critical sub-systems within a data centre environment whose continuous provision is required to guarantee uptime and availability.
When a server cabinet or additional cabinet rows are added to a datacentre, the increased demand for cooling can be accommodated for. Either by existing plant or the addition of extra CRACs or CRAHs and their associated sub-systems, maybe chiller units and heat exchangers.
Extra power could be provided by the existing UPS system or by adding additional UPS capacity.
However, the total amount of power to the building is limited by the size of the builder incomer.
Electricity will be delivered to the building from a local electrical grid connection.
This is via an incomer and distributed through LV (low voltage) switchgear and power distribution panels.
Large datacentres may also have their own direct sub-station transformer and HV (high voltage) switchgear before the LV distribution system.
The electricity supplied by the incomer may be sourced from a number of sources including:
- hydro power stations
- renewables (such as large-scale wind turbines.)
The building supply could be supplemented by on-site generated power. This can be from standby generators, solar PV, small wind turbine systems and fuel cells.
Whatever the sources of generation, the total amount of electrical power available for data centre operations is a key limiter to the overall size the data centre can grow to and it is extremely rare. It is costly to upgrade the building incomer once a data centre is operational. An alternative would be to go ‘off-the-grid’ and run on local prime power generation.
Right-Sizing And Modular Systems
The 2012 Best Practice Guide for the EU Code of Conduct on Data Centres recommends: a modular approach to data centre design, matching power and cooling systems to current IT demand. Providing ease for scalability for the data centre.
‘Modular’ defined as a standardised, flexible approach that provides for ease of expansion and repair.
It is the ease of expansion and therefore the flexibility of a modular approach that varies in relation to the size of the installation and within the range micro to large-scale datacentre. At the rack level every component is modular by default and it is relatively easy to install a new server, router or patch panel. The degree of flexibility decreases with scale and as one moves up the power distribution system towards the building incomer.
The size of a building incomer is fixed. It is extremely rare (and expensive) for this to be changed. Connected to the incomer supply, some datacentres may have a standby power generator to provide an alternative source of electricity when the mains power supply fails.
The generating set may be a standalone device or installed with a second or third generator in an N+X configuration . The generators may be considered modular, especially if supplied within their own containers and the arrangement allows one set be taken out of service for maintenance whilst still maintaining availability.
However, due to the physical scale of the installation, the design may not be truly flexible in terms of expansion. Each additional generating set added will impact existing space, pipework systems, fuel storage and delivery, exhausting and of course, electrical connection. As with a building incomer at this end of the power distribution chain, there is little scope generally for expansion and the flexibility offered by a modular approach is generally not easily available.
Uninterruptible Power Supply System
The next item to consider in the power chain is the provision for an uninterruptible power supply system. This may also be arranged in an N+X configuration and installed as a centralised or decentralised system. The term centralised is used to define a UPS system supplying site-wide power protection. De-centralised refers to the use of multiple UPS installed closer to the point of load protection i.e. at the server cabinet row level or within the server cabinets themselves.
The term ‘modular’ now becomes more relevant within the power chain when used in relation to UPS systems. At this point in the power distribution system it becomes easier to adopt the modular concept proposed by the EU Code of Conduct on Data Centres.
To meet recommendations, Use two UPS design formats:
- Standalone UPS systems
- Modular Component UPS systems
Standalone UPS Systems
The majority of UPS systems sold are standalone UPS. They may be floor standing or rack mount. Generally, connected in a parallel/redundant N+X architecture to create systems up to 6MVA. The typical individual on-line UPS ranges in size from 800kVA to as low as 1kVA. Supplied as a single cabinet. The cabinet houses all the UPS assemblies. This is the inverter, rectifier, static switch etc. Also possibly housing the battery or require connection to an external battery cabinet, battery stand, DC flywheel or fuel-cell for a source of back-up power.
Most UPS manufacturers build parallel capability into their UPS from 10kVA upwards. (with some available from 5kVA). Parallel operation provides additional resilience. When connected in an N+X arrangement, each individual UPS system is generally referred to as a UPS module. Numbered (module 1, module 2 etc). In terms of ‘right-sizing’, install a UPS system consisting of a single UPS or multiple UPS modules in an N+X arrangement.
Each UPS module is linked together through a communications interface cable to allow them to synchronise their outputs, load share and optimise their efficiency through the use of module ‘sleep-mode’ if available. Take care with a larger N+X arrangement. Ensuring adequate fault clearance in terms of matching supply cable lengths, and appropriately sized circuit breakers and protections.
This type of scalable standalone UPS system provides a flexible, modular approach for future expansion.
The ease of adding an extra UPS will depend on the:
- floor space,
- rack space,
- power distribution connections
Allow for these at the design stage.
Modular Component UPS Systems
Modular component UPS systems can offer several advantages within a specific modular component. Ranging from 10-50kVA in size. Multiple modules are connected in parallel within a single system cabinet. Ideally to achieve a higher kVA output and/or Level of N+X redundancy. The system cabinet is similar in shape and size to a server cabinet to optimise the amount of floor space required. The cabinet acts as a central hub into which UPS modules are stacked and inter-connected. The cabinet may house a battery set connected to an external battery source. This is connected to the mains power supply and to the loads it will support (directly or via some form of power distribution.)
Some UPS modules are self-contained. Installed in a server rack as standalone UPS systems. UPS manufacturer’s cabinet house a centralised back plane.
The firmware required to ensure output synchronisation between modules and the input and output supply distribution.
Within such a system, a communications cable provides inter-module co-ordination and standardised cable lengths are used to ensure adequate fault clearance.
With most modular component UPS systems, additional modules can be installed provided there is cabinet space available. UPS module size, weight and module accessibility can be issues, especially when working at height and may require a two-man installation team. Module connection is made using pre-prepared standard connection cables with ‘plug-in’ or hardwired connections. As with standalone UPS systems, additional cabinets will require extra floor space, switchgear and power distribution connections.
Modular component UPS systems can offer several advantages within a specific size range for IT operations including floor space, a more easily expandable system and the ‘hot swappable’ nature of the UPS modules themselves but at a premium price over standalone UPS systems.
There are a number of factors to consider when planning a UPS installation. The prerequisite is power quality and backup power provision. Traditionally power quality has focused on the reduction or removal of mains power problems by the UPS. This is so that the Load receives a clean, stable and regulated supply when the mains power supply is present. Backup power refers to the amount of runtime available when mains power fails. Consequently generated: standby generator, batteries, flywheel or fuel cell.
When considering UPS for critical data centres it is important to start with the EN/IEC definition of an on-line UPS: Voltage and Frequency Independent (VFI). On-Line UPS- also referred to as on-Line double-conversion or On-Line systems. ECO-mode operation by VFI systems is now on the green IT agenda in a bid to reduce operating costs through improved energy efficiency.
Most on-Line UPS offer this additional ECO-model functionality, essentially allowing the UPS to operate as a line interactive UPS when the mains power supply is relatively stable and clean.
For standalone On-line UPS, a further consideration is whether to adopt transformer-based or transformer-less design. The essential difference between the two types is the added robustness and galvanic isolation provided by a transformer -based UPS. This leads to increased reliability and improved power quality within electrically unstable environments .
Correspondingly, the lack of a transformer also provides standalone On-line transformer-less UPS with a more compact footprint and improved operating efficiency. Modular UPS are typically based on the transformer-less format and require an additional isolation transformer when installed in severe electrical environments.
It is important to note that whilst transformer-based UPS have tended to be less efficient than transformer-less designs, advances in power electronics and transformer design have reduced the efficiency margin between the two types.
Advantages and Disadvantages of Modular Transformer-less UPS
Floor Space Flexibility
There’s limited floor space within a datacentre. Required for revenue generating server racks. Expand vertically modular component UPS systems. Ensure there is room within the cabinet for additional UPS modules. Alternatively a modular component UPS system must expand horizontally with the addition of a further UPS cabinet.
This approach can save valuable space within a data centre. Advances in the design of standalone UPS systems, has also led to the introduction of smaller-footprint systems and taller cabinet designs.
What are these?
They are standalone UPS system at 80kVA. They have a comparable footprint, similarly rated modular component UPS system and in some instances, slightly less.
To provide up to 80kVA of power the modular component UPS system cabinet is sized to house up to 4x20kVA modules. The overall physical footprint is a fixed point and comparable to a standalone UPS system. The modular component UPS system can offer a more future-proof approach with right-sizing the load on day one.
Never load UPS at 100% load.
Why? As this leaves no room for expansion and can lead to increased component stress. At 60kVA load, the standalone 80kVA UPS may provide longer-term reliability than a similarly sized 60kVA modular UPS system. Whilst the modular system can be expanded through the addition of a further 20kVA module. Both have a maximum power output of 80kVA.
The basic argument for a modular component UPS system is:
Maximum operating efficiency:
Achieved by sizing the right UPS to the load size.
Maximum operating efficiency:
Achieved when operating at 80-100% of the design capacity.
The efficiency curve of modern standalone transformer-less On-line UPS has been greatly flattened.
Traditionally the maximum operating efficiency of this type of UPS was 94-95% at around 80-100% load.
For higher efficiency values, achieve a wider optimum load range. Sometimes to as little as 25% loading.
In a modular component UPS system, additional UPS modules can be added for future expansion. A standalone UPS can also be oversized to allow for this.
Either way, the same electrical connection costs are incurred.
Load sharing within the chosen system is also important and can affect operating efficiency. As UPS modules or standalone UPS are added to a parallel configuration. The load is shared between the UPS modules.
The greater the number of UPS modules the lower the overall load on each module.
This means that one 80kVA standalone UPS operating at 80% capacity, could be more efficient than 60kVA modular UPS comprising of 4x20kVA UPS modules with one module providing N+1 redundancy.
This becomes even more important in an N+2 configuration. Ensure no single system is loaded by more than 50%.
The Uptime Institute defines Tier-levels with respect to power redundancy and availability. Availability defined as: MTBF / (MTBF+MTTR).
When investigating this formula, it would be reasonable to assume that transformer-less On-line UPS, have similar or matching Mean Time Between Failure (MTBF) values.
Differences in availability figures between standalone and modular component UPS system give different MTTR or Mean Time To Repair values.
Ignore logistics for a true datum line for comparison. Consider the MTTR.
The time it takes to physically install either of these two scenarios:
- Replacement UPS module into an existing modular UPS cabinet
- The swap-out of a UPS assembly or complete disconnection and reconnection of a standalone UPS
Dependable on the cable location and access to terminations, a modular component UPS system may have a 25% lower MTTR under module failure conditions . Check the weight of the UPS module (20-40kg) and the location within the rack. (above elbow height and a safe one-person lift).
Similarly, logistics and health and safety must be considered.
The MTTR also assumes the arrival onsite of a healthy UPS module or replacement UPS assembly or complete unit. Consider entire system load banking for any replacement component, module or system. Failure to adequately load test could lead to a catastrophic failure of the entire system.
The modular component UPS system may also have a lower overall MTBF compared to a standalone UPS system. The addition of each UPS module decreases the MTBF as the component count rises. In comparison, a standalone UPS system will have a lower component count and therefore higher MTBF for the same kVA rating.
For these reasons modular UPS cabinets tend to only house up to 6-8 modules. The lower MTBF can be compounded when batteries are taken into account.
Ease of Service
Modular component UPS systems are slightly easier to service and repair in situ because a failed UPS module can be ‘hot-swapped’. To return a standalone UPS system to active service may require a board swap.
Both ‘repairs’ will rely on sound ribbon connectors, terminal connections and screw fixtures. The modular component UPS system may require a less trained technician to perform a module swap out but any system fault condition beyond this will require a UPS engineer to attend site.
Parallel / Redundant Architecture
In a modular component UPS system it is possible to build-in a level of redundancy.
Simply connect an additional UPS module. Generally limited by design to an N+1 level of redundancy.
This may not satisfy some critical data centres and their service level agreements to clients.
Limits of redundancy of a modular component UPS system are:
- The total overload rating capacity
- The alarm monitoring capabilities of the UPS cabinet itself.
Single Points of Failure
Standalone UPS systems operate a single uninterruptible power supply with no shared components other than a communications cable. Cable failure/accidental disconnection is accommodated in firmware algorithms in each UPS. Ensuring the entire system continues to function and support the load.
Modular UPS may share common components, the failure of which could result in a loss of redundancy or shutdown of the entire modular component UPS system. Common components can include:
Some modular component UPS systems can use a common three-phase AC bus with a single transfer switch.
Failure of the cabinet transfer switch could lead to a catastrophic drop of output power to the connected load.
System Control Unit:
If communication/load sharing between the UPS modules is lost, all UPS modules transferring will switch.
Common Battery Set:
A common battery can be a single point of failure.
Hence, it is especially important to perform individual battery block/string inspections in a common battery set. A failing block can weaken an entire battery string and the overall runtime available from the battery set.
Standalone UPS systems in a parallel configuration can also share a battery set and the same observations apply with regard to battery testing and inspection.
Give special consideration to modular component UPS system. Especially if a wet-cell lead acid battery is in the data centre.
Capital Costs and Return on Investment
Modular component UPS systems represent less than 5% of the global UPS market.
They have a price premium of 15-20% compared to standalone UPS systems.
What is the difference? It is mainly down to scale.
Operating efficiencies of a standalone UPS system and UPS modules within a modular component UPS system are very comparable if not near identical.
This leads to very similar Total Cost of Ownership (TCO) calculations.
What is the difference?
Only the ability of the modular component UPS system to vertically scale.
As a result, releasing floor space within a datacentre environment for revenue generating server racks, rather than critical power infrastructure. This advantage is very much within a ‘sweet spot’ size range of 20-40kVA.
They can also attract a 15-20% premium compared to a standalone UPS system.
Which type of UPS system is adopted is dependent upon where in the power distribution chain the UPS installation is to be made (power plant room or datacentre) the datacentre itself and brand reputation of the UPS manufacturer.
Most UPS manufacturers offer both standalone and modular component UPS systems.
These implement the ‘modular’ approach recommended by the ‘2012 Best Practice Guide for the EU Code of Conduct on Data Centres.’
According to Frost & Sullivan the European UPS market for modular UPS is expected to grow to £260m (US$400m) by 2017.
This demonstrates a growing acceptance of the modular component UPS system.
The growing development of small-to-medium sized datacentres and their need to rapidly expand to meet demand for Cloud-type services.
In terms of resilience, both approaches can be configured providing similar levels of availability. Modular component UPS systems have a premium price compared to standalone UPS.
However, this will fall with increased adoption and/or the introduction of a new generation of modular technology.
Whitepaper published by Riello UPS.
Search – Knowledge Base