Frequent Error when deploying Helm to GKE with Terraform

0
votes

Components

  1. GKE
  2. Helm v3
  3. Terraform

Note: The below error is raised, BUT IF i keep doing terraform apply/delete multiple times, it would somehow auto-resolve. I am making use of Google Cloud Console so there is no chacne of my Internet messing things up.

Error Type 1:

Error: Error reading ComputeNetwork "projects/foo/global/networks/bar-network": Get https://www.googleapis.com/compute/v1/projects/foo/global/networks/bar-network-e4l6-network?alt=json: dial tcp [1111:2222:4003:c03::5f]:443: connect: cannot assign requested address

Error Type 2:

Error reading Service Account "projects/foo/serviceAccounts/[email protected]": Get https://iam.googleapis.com/v1/projects/foo/serviceAccounts/[email protected]?alt=json&prettyPrint=false: dial tcp [1111:2222:4003:c04::5f]:443: connect: cannot assign requested address

Error Type 3:

Error: Error retrieving available container cluster versions: Get https://container.googleapis.com/v1beta1/projects/foo/locations/us-central1-c/serverConfig?alt=json&prettyPrint=false: dial tcp [1111:2222:4003:c03::5f]:443: connect: cannot assign requested address

Error Type 4:

Error reading instance group manager returned as an instance group URL: "googleapi: Error 404: The resource 'projects/foo/zones/us-central1-c/instanceGroupManagers/gke-bar-main-pool-8c2b8edd-grp' was not found, notFound"

I dont understand why it popsup randomly, and when I re-run same terraform apply/delete it magically works fine!

Any guidence would help.

1
terraformkubernetes-helm

1 Answers

1
votes

I had this exact same problem, and after several hours of looking at it in more detail - i think I know what is happening and how to work around it. Since implementing the following workaround, I've had 100% success rate in apply/destroy operations.

Problem:

For some reason, Terraform is accepting the AAAA (IPv6) record over the A record. You can see this in the error response as the record for the *.googleapis.com is an IPv6 address. As Google Cloud Console doesn't have IPv6 enabled, this is why you're getting this error. It seems this is a problem with Go, rather than Terraform itself based on the searches I did for similar errors.

Solution:

Short of changing the source code in Terraform, you can instead modify your /etc/hosts file to respond with an IPv4 address for each of the APIs Terraform calls. As the Cloud Shell is hosted on Google Cloud, you can use the private.googleapis.com range (199.36.153.8/30). To automate this, just put the following in your .customize_environment file in your home directory:

export APIS="googleapis.com www.googleapis.com storage.googleapis.com iam.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com"
for i in $APIS
do
  echo "199.36.153.10 $i" >> /etc/hosts
done

For reference, I created an issue in the Google provider to track it.