Kubernetes cluster disappeared from Google Container Engine

Question

I've been running an app on Kubernetes very smoothly for the past few weeks, until today I found that my Kubernetes master is no longer available.

But the application is still up, seemingly because the cluster's nodes are still available in Google Compute Engine:

(We don't use any compute instances aside from our K8s cluster.)

My gcloud CLI has all the right configuration:

$ gcloud config list
Your active configuration is: [default]

[compute]
...
[container]
use_client_certificate = True
[core]
account = {email}
disable_usage_reporting = True
project = {myproject}

Even my kubectl CLI is configured to use the cluster that once existed:

$ kubectl config current-context
gke_{myproject}_us-central1-a_{clustername}

But any kubectl call results in a timeout.

$ kubectl get pods
Unable to connect to the server: dial tcp {ipv4 address}:443: i/o timeout

gcloud and the Google Container Engine web UI both agree that there are no clusters:

$ gcloud container clusters list
*crickets*

By "crickets" do you mean the command doesn't complete, or that it completes with no output? There's a massive difference. Given the kubectl output, it looks like it's not able to talk to Google. What is shown in the Container Engine part of the console? (console.cloud.google.com/kubernetes/list) — Jon Skeet
@JonSkeet gcloud completes with no output, and the GKE web UI also shows nothing (I see a prompt to "Create a container cluster") — glifchits
Do you see any relevant actions on the activity page -- console.cloud.google.com/home/activity?project=YOURPROJECT — Eric Tune
@EricTune there has been no activity in the past 12 days... I successfully patched a deployment on this cluster and viewed logs with kubectl yesterday. — glifchits

glifchits glifchits · Accepted Answer · 2017-03-01T15:04:51

I got in touch with Google Support to solve this problem. The cluster's master instance was erroneously marked as abusive, and the master was taken down. Google was able to bring it back up. After that, there was a process to return the cluster to its normal state.

I suppose the solution here is to contact support.

Kubernetes cluster disappeared from Google Container Engine

1 Answers