How to bootstrap RBAC privileges when bringing up a GKE cluster with Terraform

7
votes

I'm bring a GKE cluster up with Terraform, which works nicely. I then want Terraform to perform some Kubernetes-level operations on the cluster (using the k8s provider) - nothing massive, just installing a couple of Deployments etc.

The problem I'm having is permissions. I'd like a neat, tidy, declarative way to make a cluster and have a set of credentials in hand that I can use short-term to do "admin" operations on it, including bootstrapping other users. I know how to make the google user that's running TF an admin of the cluster (that question comes up a lot), but that doesn't seem very nice. Not least, the k8s TF provider doesn't support clusterolebinding (Issue, partial PR) so you have to "shell out" with a local-exec provisioner to first run gcloud container clusters get-credentials and then kubectl create clusterrolebinding ....

Similarly, I don't want to set a master password, because I don't want to run with HTTP basic auth on. The nicest option looks to be the key/cert pair that's returned by the TF GKE resource, but that has a CN of "client", and that user has no power. So again, the only way to use it is to shell out to kubectl, pass it the gcloud service account credentials, and get it to add a clusterrolebinding for "client", at which point I may as well just do everything as the service account like above.

For contrast, on EKS the (AWS IAM) user that creates the cluster has cluster-admin out of the box (I assume the AWS authn provider claim's the user is in "system:masters").

My actual question here is: is there a neat, fully declarative way in Terraform to bring up a cluster and have available (ideally as output) a potent set of credentials to use and then drop? (yes I know they'll stay in the tfstate)

My options seem to be:

  • "shell out" to give TF's google ID (ideally a service account) cluster-admin (which is privilege escalation, but which works due to the gcloud authz plugin)
  • Enable HTTP basic auth and give the admin account a password, then have an aliased k8s provisioner use that to do minimal bootstrapping of another service account.
  • Enable ABAC so that "client" (the CN of the output key/cert) has infinite power - this is what I'm currently running with, don't judge me!

And I don't like any of them!

1
kubernetesgoogle-compute-engineterraformgoogle-kubernetes-engine
Within GCP, there needs to be an account created with the proper IAM roles to access the GCP resources you would like it to access. I would suggest the first option where you create a service account for Teraform and then provide it the cluster-admin priviledge within the cluster. – Jason
Same problem except I'm not as fluent in term of "shelling out" or "making the Google service account that's running TF an admin and setting it in the K8s provider". Would you mind providing guidance for that in your question? SO is kinda scarce on these problems (otherwise perfectly well articulated in your question). – b4stien

1 Answers

5
votes

I've been running into a similar problem, which has gotten particularly nasty since a recent Kubernetes issue unexpectedly disabled basic auth by default, which broke my previously-functioning Terraform configuration as soon as I tried to build a new cluster from the same config.

Finally found an answer in this SO answer, which recommends a method of using Terraform's Google IAM creds to connect to the cluster without needing the "shell out". Note that this method allows cluster permissions to be bootstrapped in Terraform with no external tooling/hacks/etc and without needing to have basic auth enabled.

The relevant part of that answer is:

data "google_client_config" "default" {}

provider "kubernetes" {
  host     = "${google_container_cluster.default.endpoint}"

  token = "${data.google_client_config.default.access_token}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.default.master_auth.0.cluster_ca_certificate)}"

  load_config_file = false
}