Terraform Throttling Route53

Question

Did anyone experienced issues with Terraform being throttled when using it with AWS Route53 records and being VERY slow?

I have enabled DEBUG mode and getting this:

2018-11-30T14:35:08.467Z [DEBUG] plugin.terraform-provider-aws_v1.36.0_x4: 2018/11/30 14:35:08 [DEBUG] [aws-sdk-go] <?xml  version="1.0"?>
2018-11-30T14:35:08.467Z [DEBUG] plugin.terraform-provider aws_v1.36.0_x4: <ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>Throttling</Code><Message>Rate exceeded</Message></Error><RequestId>REQUEST_ID</RequestId></ErrorResponse>
2018-11-30T14:35:08.518Z [DEBUG] plugin.terraform-provider-aws_v1.36.0_x4: 2018/11/30 14:35:08 [DEBUG] [aws-sdk-go] DEBUG: Validate Response route53/ListResourceRecordSets failed, will retry, error Throttling: Rate exceeded

Terraform takes >1h just to do simple Plan, something which normally takes <5 mins.

My infrastructure is organized like this:

alb.tf:

module "ALB" 
{ source = "modules/alb" }

modules/alb/alb.tf:

resource "aws_alb" "ALB" 
{ name = "alb" 
subnets = var.subnets ...
}

modules/alb/dns.tf

resource "aws_route53_record" "r53" {
  count     =  "${length(var.cnames_generic)}"
  zone_id   = "HOSTED_ZONE_ID"
  name      = "${element(var.cnames_generic_dns, count.index)}.${var.environment}.${var.domain}"
  type      = "A"

  alias {
    name    = "dualstack.${aws_alb.ALB.dns_name}"
    zone_id = "${aws_alb.ALB.zone_id}"
    evaluate_target_health = false
  }
}

modules/alb/variables.tf:

variable "cnames_generic_dns" {
  type = "list"
  default = [
    "hostname1",
    "hostname2",
    "hostname3",
    "hostname4",
    "hostname5",
    "hostname6",
    "hostname7",
     ...
    "hostname25"
      ]
}

So I am using modules to configure Terraform, and inside modules there are resources (ALB, DNS..).

However, looks like Terraform is describing every single DNS Resource (CNAME and A records, which I have ~1000) in a HostedZone which is causing it to Throttle?

Terraform v0.10.7
Terraform AWS provider version = "~> 1.36.0"

KJH KJH · Accepted Answer · 2018-12-01T02:07:17

that's a lot of DNS records! And partly the reason why the AWS API is throttling you.

First, I'd recommend upgrading your AWS provider. v1.36 is fairly old and there have been more than a few bug fixes since.

(Next, but not absolutely necessary, is to use TF v0.11.x if possible.)

In your AWS Provider block, increase max_retries to at least 10 and experiment with higher values.

Then, use Terraform's --parallelism flag to limit TF's concurrency rate. Try setting that to 5 for starters.

Last, enable Terraform's debug mode to see if it gives you any more useful info.

Hope this helps!

Terraform Throttling Route53

3 Answers