Programmatically run Kubernetes commands on a fresh GKE cluster

Question

In order to run a load test of an application I’m developing, I’m creating a Google Container Engine (GKE) cluster, create a couple of persistent disks, run a whole bunch of kubectl commands, run my benchmark, save the results, then delete the GKE cluster again.

The kubectl commands are responsible for labeling a subset of nodes of the GKE cluster (to guarantee that they only run my application, whereas the benchmark infrastructure is on the remainder of nodes) and creating secrets, replication controllers and services.

I’m looking for a way to automate all of this. I know that I could write a custom program which does all of this for me, but I’m wondering if there already is such a program.

I’ve briefly looked into https://www.terraform.io/, but it looks like that would only cover the GKE setup, not the Kubernetes resources.

Is the way to go to combine terraform and a script which runs kubectl commands? Or are there other tools for orchestrating Kubernetes?

Answer 1

Check out Cloud Deployment Manager. It allows you to declaratively create a GKE cluster and then adding resources to the cluster. I'm not sure if it will label the nodes for you. To accomplish that, one option is to create multiple node pools. Each node pool will be automatically labeled with the name of the node pool, which should help you reduce the number of manual kubectl commands that you need to run.

Answer 2

For anyone who lands here now, Terraform now (July 18) supports provisioning kubernetes resources as well, so your terraform can produce Kubernetes Deployments, Services, etc.

The only real caveat is that the provider has taken the decision to only support GA (non-alpha or beta) resources, which as of this writing is still fairly few of them (no Ingress for example).

Answer 3

I'm using terraform to create infra in different clouds, and after that Helm to provision. If are not using a K8s PaaS, I'll add to the loop Ansible.

Ansible can deploy also deploy in kubernetes, but I think it's good the extra value added by Helm.

Hope it helps.