Re-run the "program" of an external data source on every plan (refresh state) terraform

1
votes

I'm using an external data source in order to perform a cURL command using the program argument :

data "external" "curl_zip" {
    program = ["bash", "-c", "curl", ...]
}

I'm running the Terraform in a pipeline so I need to retrieve the data on every terraform plan.

It seemed to work well until I created a new resource requiring the curl to be performed. But it looks like Terraform is only refreshing and so doesn't do the program command after the first plan:

data.external.curl_zip["something.json"]: Refreshing state... [id=-]

My question is : is there a way to re-run the program argument on every plan even during refresh ?

PS : I already tried to use a null_resource instead with a local-exec, turned out to not be the solution here because (for some reason) I also need to use a archive_file data source to create zips files so my GCP app engines resource can read them, and the local-exec is being executed after the terraform apply, which doesn't work since the data source is being refreshed or created during the plan.

2
google-cloud-platformterraformterraform-provider-gcp
This is already the case. Any time the state is refreshed the program will be executed. Can you explain more clearly what you are seeing that makes you think that isn't the case? And if you can could you provide an minimal reproducible example that exhibits that behaviour please? – ydaetskcoR
Because I get an error message saying that the file I should get with curl is missing (needed by the archive file data source) + I don't see the curl output. I'll try tomorrow to provide a minimal reproducible example. – PL Sergent

2 Answers

1
votes

It seems to me like you're over-complicating things. What, exactly, are you trying to achieve? With all things Terraform it's usually better to ask how can I achieve so and so? than how can I get my Terraform code to work?.

Is the following what you're after?

data "external" "hello" {
    program = ["bash", "-c", "echo 'Hello World!' > helloworld.txt; echo -n '{\"hello\":\"world!\"}'"]
}

resource "null_resource" "world" {
  provisioner "local-exec" {
    command = "echo '${data.external.hello.result["hello"]}'"
  }
}

As you can see from the timestamps in the following output, helloworld.txt is being generated five times, once every time Terraform plan is invoked:

jdsalaro$ for i in {1..5} ;do terraform plan; ls -lah --full-time helloworld.txt ;done \
| grep helloworld.txt | cut -d ' ' -f 7,9

00:04:18.610304219 helloworld.txt
00:04:19.902246088 helloworld.txt
00:04:21.226186506 helloworld.txt
00:04:22.574125835 helloworld.txt
00:04:23.886066774 helloworld.txt

I uploaded the whole example here just in case.

0
votes

Update

Thank you very much for your response and sorry for not getting back to you earlier. So for some reason I had a hard time using the data source, and it seems that after an apply the data source was saved into the state : terraform state list. So the following plan did not recreate the data source that I was using to execute the curl.

So I got back to the null_resource solution. And it's a little bit complicated, but previously I had a problem with the curl request that have made the zip unusable for my GAE resource. So I had to use an archive_file on top of the curl. But this couldn't work because a data source is being treated during the plan and the local-exec of the null_resource is being executed during the apply.

Anyway so I fixed my curl so I don't need the archive_file data source. I also needed to change the interpreter to ["/bin/bash", "-c"] to make this work. Moreover, I used a trigger to always run the curl during each apply.

Here is my resource :

resource "null_resource" "curl_zip" {
    for_each = local.json_data
    provisioner "local-exec" {
        command = "curl -H 'API_KEY' -sLo ./path/to/zip"
        interpreter = ["/bin/bash", "-c"]
    }

    triggers = {
        always_run = timestamp()
    }
}