The Domino Effect: AWS Outage Caused by DNS Automation Race Condition

Following the massive service outage that occurred on October 20 in its US-EAST-1 (Northern Virginia) region, AWS has formally released the findings of its internal investigation, tracing the root cause to DynamoDB’s DNS automation module. According to AWS, two automated modules—the DNS Planner and the DNS Enactor—manage DynamoDB’s DNS operations, with three independent Enactor instances running across separate Availability Zones to ensure high availability.

The sequence saw Enactor A lagging on updates while Enactor B rapidly completed its changes, and a subsequent cleanup by Enactor B mistakenly treating Enactor A’s ongoing deployment as obsolete, erasing all IP addresses for US-EAST-1 service nodes and leaving DNS records empty.

The incident disrupted 142 AWS services and thousands of customers, and took roughly three hours to resolve the core DNS malfunction, though cascading effects endured for over a dozen hours due to interdependencies such as the Droplet Workflow Manager and downstream services like NLB and AWS Lambda. As a corrective measure, AWS suspended DynamoDB DNS Planner and DNS Enactor automation globally pending safety audits and race-condition fixes.