Data Center Resilience: Design, Operations and AI Loads

Data center resilience is a facility's ability to keep serving workloads through equipment failures, power loss, cooling faults, network cuts and human error, and to recover quickly when it cannot. It is built in two places: in the design (redundant power, cooling and network paths) and in daily operations (tested procedures, trained people, disciplined maintenance). For AI and HPC, where racks draw far more power than they did a few years ago, both halves matter more than they used to.

This guide is written for CTOs, IT managers and infrastructure leads who must decide how much resilience a deployment needs, where the real risks sit, and what to do about them.

Dormedia

dormedia.ai

Build Resilience Into AI Infrastructure

Dormedia helps organizations design, deploy and scale tailored HPC and AI infrastructure for demanding workloads and complex deployments.

Explore Dormedia’s solutions

TL;DR

Resilience is not the same as redundancy. Redundant components are the foundation, but resilience also covers procedures, monitoring, maintenance and recovery.
Power is still the most common source of serious data center outages, and many serious outages are preventable with better management, processes and configuration.
AI and HPC density changes the failure profile: liquid cooling, higher-voltage distribution and cooling ride-through become resilience questions, not just efficiency ones.
Match the level of protection to the cost of downtime. Tier and Rated classifications give you a common language for that decision.
Write the recovery plan, test it, and re-test it at least once a year.

What data center resilience means
What actually takes data centers down
Designing for resilience: power, cooling, network, facility
Redundancy, failover and how much is enough
What AI and HPC change
Operations: the half that preserves resilience
Making resilience part of the project from the start
Work with Dormedia on resilient HPC and AI infrastructure
FAQ
Sources

What data center resilience means

A data center is resilient when a single failure, planned or unplanned, does not become an outage for the people depending on it. One widely used definition describes it as the ability of a server, network, storage system or whole facility to recover quickly and keep operating after an equipment failure, power outage or other disruption.

Two distinctions save a lot of confusion in planning meetings.

Resilience versus redundancy. Redundancy means spare parts: a second UPS, a second cooling loop, a second network path. Resilience is what you get when those spares are correctly designed, maintained, monitored and rehearsed. You can have redundancy without resilience, for example when two power paths share one weak transfer switch or when nobody has tested the failover in two years.

Data center resilience versus data resilience. Data resilience is about the information: backups, replication, recovery from corruption, ransomware or accidental deletion. Data center resilience is about the physical and operational platform that keeps compute, storage and networking alive. IT resilience rolls both into business continuity planning. You need all three, but they are owned by different people and tested in different ways.

What actually takes data centers down

Before spending on protection, look at where failures really come from. The numbers below come from the Uptime Institute's outage research, the most cited body of operator-reported incident data.

Power leads the list. In the Uptime Institute Annual Outage Analysis 2026, power issues accounted for 45% of respondents' most recent impactful outages in 2025, down from 54% in 2024. That reverses an upward trend seen since 2020. The most frequently cited causes are UPS failures, followed by transfer switch and generator issues.

Networks and fiber are rising. The same report finds fiber and cable incidents were the leading cause of publicly reported IT service outages in 2025, with the largest increase against the 2020 to 2025 average. Subsea cables and terrestrial routes can fail through sabotage, accidental damage, theft or environmental factors. The 2024 Resiliency Survey summary adds that network issues are the largest single cause of IT service outages and that third-party provider problems have crept up to nearly one in ten outages.

Cooling is the second physical risk. Data Center Knowledge summarizing Uptime data put cooling problems at 19% of outages in a 2024 article, behind power.

Most outages are avoidable. The 2024 survey summary reports that four in five respondents said their most recent serious outage could have been prevented with better management, processes and configuration. It also reports that 54% of respondents said their latest significant outage cost more than $100,000, and 16% said more than $1 million. The same summary notes that overall outage frequency and severity are actually decreasing, with 10 to 20 high-profile events a year worldwide.

The practical reading: spend first on power-chain quality and testing, network path diversity, cooling continuity and procedures. Exotic scenarios come after.

Designing for resilience: power, cooling, network, facility

Power

A resilient power chain has no single point of failure between the utility feed and the rack. That means independent paths, maintainable switchgear, generators that start and carry load, and UPS systems that are tested under load rather than assumed. Given that UPS, transfer switch and generator faults are the leading outage causes cited above, these are the components deserving the most testing time.

Standby generator in a data center plant room

For larger and AI-focused sites, ASHRAE's AI Data Center Energy Performance Framework says resilient infrastructure needs careful planning of both electrical and mechanical systems for 24/7 performance. It also describes microgrids as critical for operational resilience, allowing islanding during grid difficulties, controlled resynchronization and black-start capability under IEEE 2030.7.

Cooling

Cooling resilience is where AI and HPC change the rules. ASHRAE TC 9.9's liquid cooling resiliency guidance warns that loss of cooling can be catastrophic at extreme chip powers. It cites manufacturer experience of as much as a 27°C (48.6°F) jump in coolant temperature when CDU pumps changed over without active redundancy, and research indicating processors can throttle within as little as 23 seconds of a CDU pump failure.

The same document recommends:

Using a coolant distribution unit (CDU) and active redundancy on loop pumps.
Increasing the system's thermal inertia.
Running transient modeling of the cooling loops.
Considering a lower S-Class in high-density zones.
Deploying UPS capacity that keeps the cooling loop running when utility power is lost.
Using thermal energy storage to supply cooling water if a chiller plant loses power.

It also stresses that designers should care about the rate of change during recovery, not only during the failure itself.

Network

Physical path diversity matters more than the number of carriers on a contract. Two providers that share one conduit or one cable landing are one provider for failure purposes. Ask for route maps, entry points and where each carrier's paths converge. Because third-party providers account for a growing share of outages, treat carriers, colocation operators and hardware suppliers as part of the resilience design.

Facility

Fire detection and suppression, lightning protection, physical security and site selection all belong in the same plan. Fire is a good example of a risk that gets harder as density rises, and data center fire protection engineering is a specialty worth involving early, not after the room is built.

Redundancy, failover and how much is enough

Redundancy comes in degrees, and standards give them names. Two frameworks are in common use.

The Uptime Institute Tier Classification System defines Tier IV as fault tolerant: several independent, physically isolated systems provide redundant capacity and distribution paths, so planned and unplanned events do not affect the environment. IT equipment must have a fault-tolerant power design to be compatible, and Tier IV requires continuous cooling.

The TIA-942 standard uses four ratings:

Rating	Name	What it means
Rated-1	Basic	No redundancy requirement
Rated-2	Redundant Capacity Components	Spare capacity components
Rated-3	Concurrently Maintainable	Any single power, cooling or cabling component can be serviced without disrupting IT operations
Rated-4	Fault Tolerant	Redundant systems sustain operations automatically through an equipment failure, without human intervention

TIA states that facility certification is valid for 3 years, with annual surveillance audits.

One detail from Uptime's continuous cooling paper is directly relevant to HPC: Tier IV is the only Tier that requires continuous cooling, and Uptime recommends continuous cooling at densities beyond 4 kW per rack regardless of Tier. Most AI racks are far past that threshold.

Choosing a level

There is no universally correct tier. Work backward from the cost of an hour of downtime:

Estimate hourly cost for each workload: lost revenue, idle GPU time, missed training runs, contractual penalties.
Separate workloads that can pause (batch training with checkpoints) from those that cannot (customer-facing inference, trading, clinical systems).
Put the second group in the higher-resilience space and design the first group for fast, cheap restart.
Check that your recovery time targets are met by the design, not just by the brochure.

Mixed approaches are normal. A checkpointing training cluster can tolerate a different design than the control plane and storage it depends on.

What AI and HPC change

Density is the headline. McKinsey's 2025 report on AI data center infrastructure describes mainstream IT rack deployments moving from under 30 to 50 kW per rack to 70 to 100 kW per rack, with inference at 20 to 40 kW per rack and training up to 100 kW. It states that air cooling is no longer sufficient at these densities and a shift to liquid cooling is required.

AI rack power density ranges and cooling shift

The IEEE heterogeneous integration chapter on HPC and AI notes data center-class AI packages with thermal design power of 500 to 2,300+ watts, and flags power distribution to racks and servers as a further challenge, along with the water use of cooling systems.

For resilience planning, that translates into four concrete changes:

Shorter ride-through. Less thermal mass per kilowatt means less time to react. Cooling continuity moves from a comfort feature to a protection for the hardware itself.
New failure points. CDUs, manifolds, quick disconnects and coolant chemistry become parts of the failure model.
Power chain upgrades. Higher-voltage distribution and larger power blocks mean fewer, bigger components, so each failure affects more load.
Longer lead times. Grid connections for large loads can take years to secure, and specialist cooling and power gear has its own delivery times. A resilient plan includes supply chain resilience: spares, alternate vendors, and a realistic delivery schedule.

Where the site itself is still being chosen, the land and power requirements for AI data centers are worth reviewing alongside the resilience design, since power availability shapes both.

Our perspective

We have worked on infrastructure projects since 2011 and have been deeply engaged in HPC and AI since 2017. The pattern we see is that resilience problems rarely start in the data hall. They start in the project: a configuration that was never validated end to end, a component substituted late, a shipment held at customs, a spare that nobody sourced.

That is why we treat resilience as a project discipline as much as a design one. Our work covers design, sourcing, deployment, scaling and ongoing support, and our global vendor and logistics network exists so that the equipment you specified is the equipment that arrives, on a timeline you can plan around, with tax and tariff questions handled before they become delays.

For AI and HPC clients, we recommend deciding the resilience target first, then specifying servers, components and facility requirements to meet it, instead of buying hardware and discovering the gaps later. Customized configurations, a clear spares strategy and a tested support model are far cheaper to define on day one than to retrofit after the first incident.

Dormedia

Operations: the half that preserves resilience

Design gives a facility the capability. Operations decide whether it still exists in year five. The research on outages points the same way: the share of serious outages attributed to management, process and configuration is large.

What good operational resilience looks like in practice:

Documented procedures. Standard, maintenance and emergency operating procedures reduce improvisation when something goes wrong at 3 a.m.
Maintenance without risk. Concurrent maintainability only helps if technicians follow a method of procedure that has been reviewed in advance.
Monitoring that covers the cooling loop. With seconds-scale thermal response in liquid-cooled racks, alerts need to be fast and routed to someone who can act.
Configuration control. Many avoidable outages come from settings drifting away from the design.

Writing and testing the recovery plan

FM Global's Data Sheet 5-32, a property-protection standard widely used by insurers and operators in the US, expects a detailed written disaster recovery plan that is reviewed and tested annually. It also calls for an Equipment Contingency Plan covering electrical and cooling equipment breakdowns, and a Service Interruption Plan that systematically assesses each utility and the impact of losing it.

Outside the US, regulators are specific too. Singapore's IMDA advisory guidelines, issued in February 2025, ask data center operators to run a business continuity management system on a Plan-Do-Check-Act loop, perform business impact analysis to find critical functions and maximum tolerable disruption time, and put in place measures such as redundant power, lightning protection, fire detection and suppression, and defenses against ransomware. Those rules apply in Singapore, but the method is a sound template anywhere.

A simple sequence works for most teams:

Map dependencies. List power, cooling, network, storage, vendors and people behind each critical service.
Set targets. Define the maximum tolerable disruption and recovery time for each service.
Write the plans. One for equipment failure, one for loss of each utility, one for site loss.
Test in escalating steps. Start with a walkthrough, move to a planned failover, and finish with an unannounced drill once the team is ready.
Fix and repeat. Record what failed, change the plan or the design, and schedule the next test.

Making resilience part of the project from the start

Resilience is cheapest when it is specified at the beginning: in the server and component configuration, the facility requirements, the vendor choices and the delivery plan. Migrations are a moment of unusual risk, so our data center migration planning guide covers how to move workloads without losing availability. Because suppliers are part of your risk surface, our guide to global vendor management for HPC and AI teams covers how to keep the supply side dependable.

Work with Dormedia on resilient HPC and AI infrastructure

We help organizations design, source, deploy and scale HPC and AI infrastructure, from customized server and component configurations and data center solutions to global shipping, tax and tariff compliance, project management, strategic advisory and ongoing support. If you are planning a deployment and want resilience built into the specification, delivery schedule and support model, talk to us about your project.

Dormedia

Need hardware in the meantime? Browse our available inventory.

FAQ

What is the difference between data center resilience and redundancy?

Redundancy is having spare components or paths. Resilience is the outcome: the facility keeps serving workloads and recovers quickly when something fails. Redundancy is necessary for resilience, but tested procedures, maintenance, monitoring and recovery planning are what make the spares work when needed.

What is the most common cause of data center outages?

Power. The Uptime Institute's 2026 analysis attributes 45% of respondents' most recent impactful outages in 2025 to power issues, with UPS failures cited most often, followed by transfer switch and generator problems. Network and fiber incidents are the leading cause of publicly reported IT service outages.

How often should a disaster recovery plan be tested?

At least annually. FM Global's Data Sheet 5-32 expects a written disaster recovery plan to be reviewed and tested every year. Many teams add more frequent partial tests, such as planned failovers of individual systems, and re-test after any major change to the facility.

Do AI workloads need a higher resilience tier?

Not automatically, but they raise the stakes of cooling and power failures. Racks at 70 to 100 kW or more with liquid cooling have little thermal margin, so continuous cooling and well-tested power paths matter. Workloads that checkpoint can tolerate more interruption than live inference, so tier decisions should follow workload cost of downtime.