Terraform: How to run remote-exec more than once?

Question

I have noticed that terraform will only run "file", "remote-exec" or "local-exec" on resources once. Once a resource is provisioned if the commands in a "remote-exec" are changed or a file from the provisioner "file" is changed then terraform will not make any changes to the instance. So how to I get terraform to run provisioner "file", "remote-exec" or "local-exec" everytime I run a terraform apply?

For more details:

Often I have had a resource provisioned partially due to an error from "remote-exec" causes terraform to stop (mostly due to me entering in the wrong commands while I'm writing my script). Running terraform again after this will cause the resource previously created to be destroyed and force terraform to create a new resource from scratch. This is also the only way I can run "remote-exec" twice on a resource... by creating it over from scratch.

This is really a drawback to terraform as opposed to ansible, which can do the same exact job as terraform except that it is totally idempotent. When using Ansible with tasks such as "ec2", "shell" and "copy" I can achieve the same tasks as terraform only each of those tasks will be idempotent. Ansible will automatically recognise when it doesn't need to make changes, where it does and because of this it can pick up where a failed ansible-playbook left off without destroying everything and starting from scratch. Terraform lacks this feature.

For reference here is a simple terraform resource block for an ec2 instance that uses both "remote-exec" and "file" provisioners:

resource "aws_instance" "test" {

count = ${var.amt}
ami = "ami-2d39803a"
instance_type = "t2.micro"
key_name = "ansible_aws"
tags {
  name = "test${count.index}"
}

#creates ssh connection to consul servers
connection {
  user = "ubuntu"
  private_key="${file("/home/ubuntu/.ssh/id_rsa")}"
  agent = true
  timeout = "3m"
} 

provisioner "remote-exec" {
  inline = [<<EOF

    sudo apt-get update
    sudo apt-get install curl unzip
    echo hi

  EOF
  ]
}

#copying a file over
provisioner "file" {
  source = "scripts/test.txt"
  destination = "/path/to/file/test.txt"
}

}

Answer 1

Came across this thread in my searches and eventually found a solution:

resource "null_resource" "ansible" {

  triggers {
    key = "${uuid()}"
  }

  provisioner "local-exec" {
  command = "ansible-playbook -i /usr/local/bin/terraform-inventory -u ubuntu playbook.yml --private-key=/home/user/.ssh/aws_user.pem -u ubuntu"
  }
}

You can use uuid(), which is unique to every terraform run, to trigger a null resource or provisioner.

Answer 2

The Terraform docs on provisioning explicitly that it considers the use of provisioners for basic bootstrapping as a once only task and that it should not be used as a replacement for a proper configuration management tool such as Ansible:

Provisioners are run only when a resource is created. They are not a replacement for configuration management and changing the software of an already-running server, and are instead just meant as a way to bootstrap a server. For configuration management, you should use Terraform provisioning to invoke a real configuration management solution.

and

If a resource successfully creates but fails during provision, Terraform will error and mark the resource as "tainted." A resource that is tainted has been physically created, but can't be considered safe to use since provisioning failed.

When you generate your next execution plan, Terraform will remove any tainted resources and create new resources, attempting to provision again. It does not attempt to restart provisioning on the same resource because it isn't guaranteed to be safe.

Terraform does not automatically roll back and destroy the resource during the apply when the failure happens, because that would go against the execution plan: the execution plan would've said a resource will be created, but does not say it will ever be deleted. But if you create an execution plan with a tainted resource, the plan will clearly state that the resource will be destroyed because it is tainted.

Provisioning is important for being able to bootstrap instances. As another reminder, it is not a replacement for configuration management. It is meant to simply bootstrap machines. If you use configuration management, you should use the provisioning as a way to bootstrap the configuration management utility.

Consider the provisioners as akin to an EC2 user data script in that it only runs once on creation and if it fails then you need to destroy the instance and try again.

The benefits of this is that Terraform doesn't need to have any knowledge of how to make the changes idempotent on the operating system as Terraform works on a level higher than the instance itself and more at provisioning an entire data centre.

If you need more flexibility than this then consider either using Terraform to call a configuration management system to properly provision the instance (and then allowing for retries in that if it fails, decoupled from the Terraform provisioning stage) or using an orchestration tool such as Jenkins to wrap both Terraform and an alternative configuration management tool such as Ansible.

One other option is to go more along the route of immutable infrastructure and use Packer to create an AMI using Ansible or some other tool and then just use Terraform to deploy the AMI as is without then needing to further provision the instance.

Answer 3

You can use the taint command to mark a resource as tainted, forcing it to be destroyed and recreated on the next apply.

Answer 4

Similar answer to Chris Holmes but using timestamp and you will need to remove the ${""} around the UUID or timestamp() from Chris's answer as nowadays you will get a message of:

Warning: Interpolation-only expressions are deprecated

Unless you are using Terraform 0.11 or earlier.

resource "null_resource" "ansible" {

  triggers = {
    always_run = timestamp()
  }

  provisioner "local-exec" {
  command = "ansible-playbook -i /usr/local/bin/terraform-inventory -u ubuntu playbook.yml --private-key=/home/user/.ssh/aws_user.pem -u ubuntu"
  }
}