Terraform apply access denied error

Question

So I was upgrading Terraform from 0.9.5 to 0.9.6 and I am now getting the following error when I run a jenkins job on a build slave with IAM permissions attached:

terraform096 apply -var db_snap_stamp=171120171217 -var db_snapshot=rds-dev-13102017 -var-file=env.tfvars -no-color
Error loading state: AccessDenied: Access Denied
status code: 403, request id: 288766CE5CCA24A0, host id: FOOBAR

The jenkins job does run terraform init before hand and on my local test server I am not seeing the error. On the local test server I am using an aws credentials file.

I have had a look through the release notes for 0.9.6 but I can't see which of the changes could be causing this ( https://github.com/hashicorp/terraform/issues/14423 maybe?).

Any ideas?

UPDATE

I turned on terraform debug and found that the 403 was happening on a s3 list object. The IAM role in use allows this in 0.9.5 but NOT in 0.9.6 - I tried giving the role admin access but no change:

-----------------------------------------------------
2017/11/17 15:01:47 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects 
Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 17 Nov 2017 15:01:47 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-west-2

The S3 bucket in question does use KMS encryption but all that is set up in the init run prior:

terraform096 init -backend=true -get=true -input=false -backend-config="bucket=${BUCKET}" -backend-config="key=${ENV}.tfstate" -backend-config="region=eu-west-2" backend-config="profile=${AWS_PROFILE}" -backend-config="encrypt=true" -backend-config="kms_key_id=${KMS}"

Did 0.9.5 run successfully on the same slave (not just master) before? — James Thorpe
Yes it did - have a few version of Terraform on that slave for testing "terraform095 apply foo" works but "terraform096 apply foo" throws the error. — SnazzyBootMan
I have upgraded Terraform all the way up to 0.10.8 and when using KMS encrypted S3 buckets and S3 endpoints this error persists. Switching back to Terraform 0.9.5 without changing any other settings works as expected and builds the environment. — SnazzyBootMan

SnazzyBootMan SnazzyBootMan · Accepted Answer · 2017-11-21T09:06:05

So I raised this as an issue with Hashicorp and this does look like a bug - if anyone wants to keep an eye on the progress have a look here:

https://github.com/hashicorp/terraform/issues/16710

Terraform apply access denied error

1 Answers