Terraform - having timing issues launching EC2 instance with instance profile

Question

I'm using Terraform to create my AWS infrastructure.

I've a module that creates an "aws_iam_role", an "aws_iam_role_policy", and an "aws_iam_instance_profile" and then launches an EC2 Instance with that aws_iam_instance_profile.

"terraform plan" works as expected, but with "terraform apply" I consistently get this error:

* aws_instance.this: Error launching source instance: InvalidParameterValue:   IAM Instance Profile "arn:aws:iam::<deleted>:instance-profile/<deleted>" has no associated IAM Roles

If I immediately rerun "terraform apply", it launches the EC2 instance with no problem. If I run a "terraform graph", it does show that the instance is dependent on the profile.

Since the second "apply" is successful, that implies that the instance_policy and all that it entails is getting created correctly, doesn't it?

I've tried adding a "depends_on" and it doesn't help, but since the graph already shows the dependency, I'm not sure that is the way to go anyway.

Anyone have this issue?

Answer 1

Race conditions are quite common between services - where state is only eventually consistent due to scale. This is particularly true with IAM where you will often create a role and give a service such as EC2 a trust relationship to use the role for an EC2 instance, but due to however IAM is propogated throughout AWS, the role will not be available to EC2 services for a few seconds after creation.

The solution I have used, which is not a great one but gets the job done, is to put the following provisioner on every single IAM role or policy attachment to give the change time to propagate:

resource "aws_iam_role" "some_role" {
    ...
    provisioner "local-exec" {
    command = "sleep 10"
}

Answer 2

In this case you may use operation timeouts. Timeouts are handled entirely by the resource type implementation in the provider, but resource types offering these features follow the convention of defining a child block called timeouts that has a nested argument named after each operation that has a configurable timeout value. Each of these arguments takes a string representation of duration, such as "60m" for 60 minutes, "10s" for ten seconds, or "2h" for two hours.

resource "aws_db_instance" "example" {
  # ...

  timeouts {
    create = "60m"
    delete = "2h"
  }
}

Ref: https://www.terraform.io/docs/configuration/resources.html