AWS Route53 DNS Service Interruption – Not Updating Issue “Rate Exceeded”

For those that say AWS cloud services are 100% reliable, one only needs to look at recent outages to see that while reliable, they are nowhere near 100%. One issue currently active is that the AWS Route53 DNS changes are not even being accepted. AWS does have an active service alert on the “Open issues” dashboard, but this can be a major impact. If you need to change DNS entries to change load balancers, servers or even just to get developers started on developing a new website that activity has ground to a halt. If you have AWS Route53 DNS changes scripted, your scripts may be erroring out right now.

To test the outage firsthand, we used Route53 to change the DNS record of a development website at 7:50AM PDT as we did earlier in the evening and we immediately saw a warning message when trying to create a new A record. An A record is a very basic DNS building block that allows machines to map names (e.g. servethehome.com) to IP addresses. Here is the message you may be getting right now (we tried several times even using the AWS web console):

This has a large impact also for users of solutions such as Let’s Encrypt as you often will need to point the service to a server with an active DNS record to validate ownership. If your new domain DNS is hosted through AWS Route53, your normal workflow for development is likely not working.

AWS Route53 Open issue Log Snapshot

Here is a snapshot of the Open issues tracker as of this writing.

Slow propagation times

04:44 PM PDT We are investigating slow propagation of DNS edits to the Route 53 DNS servers. This does not impact queries to existing DNS records.

05:11 PM PDT We continue to investigate slow propagation of DNS edits to the Route 53 DNS servers. This does not impact queries to existing DNS records.

06:34 PM PDT We have identified root cause of the slow propagation of DNS edits to the Route 53 DNS servers and are working towards recovery. This does not impact queries to existing DNS records.

07:40 PM PDT We continue to experience slow propagation times and continue to work towards full recovery. This does not impact queries to existing DNS records.

10:12 PM PDT We continue to work on resolving the slow propagation times. Requests to the ChangeResourceRecordSets API are currently being throttled. Queries to existing DNS records remain unaffected.

12:22 AM PDT While changes are propagating, we continue to work through the backlog of pending changes that have accumulated. We expect full recovery to take several more hours. We have also throttled ChangeResourceRecordSets API call. Queries to existing DNS records remain unaffected

01:40 AM PDT Record changes are slowly propagating, while we work through the backlog of pending changes that have accumulated. We still expect full recovery to take several more hours. We are continuing to throttle ChangeResourceRecordSets API calls. Queries to existing DNS records remain unaffected.

03:01 AM PDT Record changes are still propagating, while we work through the backlog of pending changes that have accumulated. We expect full recovery to take several more hours. We are continuing to throttle ChangeResourceRecordSets API calls. Queries to existing DNS records remain unaffected.

04:07 AM PDT All outstanding DNS record changes have completed propagating. ChangeResourceRecordSets API calls are still being throttled. Queries to existing DNS records remain unaffected.

05:12 AM PDT ChangeResourceRecordSets API calls are still being throttled while we continue to recover. Queries to existing DNS records remain unaffected.

07:12 AM PDT We continue to throttle some ChangeResourceRecordSets API calls as we make progress towards recovery. Queries to existing DNS records remain unaffected.

Event data