An AWS outage exposed the dependencies inside a region
The AWS disruption left many running systems intact while impairing the services needed to observe, replace or expand them. Recovery architecture had to account for that difference.
On 7 December 2021, a network congestion event in AWS's Northern Virginia region exposed an important distinction: infrastructure could keep running while the mechanisms needed to manage it were impaired. Existing EC2 instances continued operating, but launching replacements became difficult. Existing DNS answers remained available while changes to DNS records were blocked by API problems.
AWS traced the disruption to connections between two internal network environments. For software companies selling dependable service on rented infrastructure, these dependencies had direct commercial consequences. A diagram showing redundant servers could leave a critical recovery path unexamined.
An attempt to scale created persistent congestion
At 7:30 a.m. Pacific Standard Time, automated capacity scaling triggered unexpected behaviour among clients on an internal AWS network. A surge of connections overwhelmed devices linking that network with the main AWS network. Delays generated additional connection attempts and retries, keeping the devices congested. AWS subsequently said a latent problem prevented clients from backing off adequately, despite the code having operated for years.
The internal network hosted monitoring, internal DNS, authorisation services and parts of the EC2 control plane. Its impairment therefore affected several products, even as the main network and many customer applications continued to operate. The region had suffered a partial failure whose consequences extended beyond whether a customer's virtual machines were still running.
Healthy resources could become difficult to replace
The control plane is the collection of services used to create and manage resources. AWS reported that EC2 APIs suffered elevated errors and latency from 7:33 a.m.; launching new instances recovered at 2:40 p.m. Products that depended on new EC2 capacity, including RDS and EMR, could consequently encounter provisioning failures. Existing load balancers remained healthy while provisioning and instance registration were delayed.
Scroll to view all columns
| Comparison | Reported to remain available | Reported impairment |
|---|---|---|
| EC2 | Running instances | APIs for launches and resource descriptions |
| Elastic Load Balancing | Existing load balancers | New provisioning and registration of additional instances |
| Route 53 | Existing DNS entries and query answers | APIs used to change DNS entries |
| Containers | Existing tasks and pods | Restarting or replacing instances through affected control-plane APIs |
Service behaviours reported by AWS. Application availability also depended on how each customer used the affected services.
The distinction extended to containers. Existing tasks and pods continued to operate, but a container that stopped could be difficult to restart because its recovery depended on affected APIs. A startup that normally tolerates individual failures by replacing workers may lose that protection during an incident affecting the management plane. Keeping capacity available, reducing optional work and pausing disruptive deployments can help, although none guarantees continuity if another required service is also impaired.
The repair process had its own dependencies
Congestion impaired real-time monitoring available to AWS operators, who initially relied on logs. Moving internal DNS traffic away from congested paths resolved DNS errors at 9:28 a.m., but did not eliminate the wider congestion. Deployment systems were also affected, slowing remedial work. AWS described a further constraint: operators had to make changes carefully because many customer workloads were still functioning.
Customers faced monitoring delays in CloudWatch, and AWS said a small amount of monitoring data might be missing. The service health dashboard's failover tooling was impaired, as was the ability to open support cases. A young software company needs to consider the same dependencies in its own incident process. If telemetry, authentication and the status page rely on the same impaired environment, a team may lose both operational visibility and its route to explain the disruption to customers.
Network restoration was one milestone among several
AWS reported that the affected networking devices had fully recovered by 2:22 p.m. That did not end every service impact. Some services needed to recycle unhealthy capacity, process queued work or restore their capacity pools. API Gateway largely recovered at 4:37 p.m., with some errors continuing afterwards. EventBridge delivery latency remained elevated until 6:40 p.m. while a backlog was processed.
| Period or category | minutes after 07:30 PST |
|---|---|
| Network devices fully recovered | |
| New EC2 launches recovered | |
| STS fully recovered | |
| API Gateway largely recovered | |
| EventBridge delivery latency recovered |
Calculated from AWS's reported times: 14:22, 14:40, 16:28, 16:37 and 18:40 PST. These are elapsed times to different recovery milestones, not comparable full-service downtime estimates; API Gateway still had some errors after its milestone.
Source1
The chart measures elapsed time from the common 7:30 a.m. onset to selected recovery milestones. Symptoms and recovery criteria differed by product, so the bars do not represent comparable outage durations. A provider's infrastructure recovery announcement may arrive before a dependent application's queue has drained or before end users see normal service.
Recovery design began with what the business could tolerate
An earlier AWS architecture article set out the economic choices. It distinguished the maximum acceptable time to restore service, the recovery time objective, from the maximum acceptable age of recovered data, the recovery point objective. Tightening either generally required additional resources and operational complexity. Its strategies ranged from backup and restore through partially provisioned standby environments to active service across multiple regions.
A second region is therefore only part of a recovery plan. A backup can protect data without providing immediate processing capacity. A pilot light environment still requires resources to be started; warm standby may require scaling before it can carry production demand. Active service in multiple regions needs continuing data coordination and operational attention. The business must decide which interruption it can tolerate, then test whether its chosen dependencies allow recovery within that limit.
Infrastructure startups can help customers identify hidden dependencies, validate recovery procedures and establish what worked in practice. The purchasing case should include the customer's additional cloud spend and engineering time. A successful demonstration of failover is valuable only if the process remains supportable as the customer's data, permissions and application architecture change.
Investment implications
- An application can survive an infrastructure fault yet fail when it needs replacement capacity or a management API.
- Monitoring and customer communications need consideration in the same failure scenarios as the production service.
- Recovery spending should follow explicit tolerances for downtime and data loss, with the chosen approach exercised in practice.
Sources
Published
