kube-apiserver refusing external connections on port 443

Question

I have a kubernetes cluster provisioned using kube-aws. I added two additional controllers and clustered etcd across the three controller nodes. The etcd cluster seems (mostly) healthy and works when I set and get values on the different nodes.

I added the new etcd servers to /etc/kubernetes/manifests/kube-apiserver.yaml:

--etcd-servers=http://10.0.0.50:2379,http://10.0.1.50:2379,http://10.0.2.50:2379

Now, the kube-apiserver refuses connections on secure port 443.

The kubelet logs contain entries like this...

E0617 02:07:17.488289   30972 pod_workers.go:138] Error syncing pod 
b394008e8164, skipping: failed to "StartContainer" for "etcd-container" with 
CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd-container 
pod=etcd-server-ip-10-0-0-50.eu-west-1.compute.internal_default(b394008e8164)"

and this...

E0617 02:03:00.542951   30972 event.go:193] Server rejected event 
'&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, 
ObjectMeta:api.ObjectMeta{Name:"etcd-server-ip-10-0-0-50.eu-
west-1.compute.internal.1458b", GenerateName:"", Namespace:"default", 
SelfLink:"", UID:"", ResourceVersion:"1189", Generation:0, 
CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, 
loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), 
DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), 
Annotations:map[string]string(nil)}, 
InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"default", 
Name:"etcd-server-ip-10-0-0-50.eu-west-1.compute.internal", 
UID:"b394008e8164", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, 
Reason:"FailedSync", Message:"Error syncing pod, skipping: failed to 
\"StartContainer\" for \"etcd-container\" with RunContainerError: 
\"runContainer: API error (500): mkdir /usr/local/openssl: read-only file 
system\\n\"\n", Source:api.EventSource{Component:"kubelet", Host:"ip-
10-0-0-50.eu-west-1.compute.internal"}, 
FirstTimestamp:unversioned.Time{Time:time.Time{sec:63601711997, nsec:0, 
loc:(*time.Location)(0x3b1a5c0)}}, 
LastTimestamp:unversioned.Time{Time:time.Time{sec:63601725780, 
nsec:522501285, loc:(*time.Location)(0x3b1a5c0)}}, Count:7, Type:"Warning"}': 
'events "etcd-server-ip-10-0-0-50.eu-west-1.compute.internal.1458b" not 
found' (will not retry!)

Querying the kube-apiserver locally on port 8080 like so...

$ curl http://localhost:8080/api/v1/namespaces

... returns a valid response.

But it refuses any attempt on the secure port.

Can anyone point me the right direction?

lander2k2 lander2k2 · Accepted Answer · 2016-06-21T20:48:11

For anyone that finds this question having trouble with the kube-apiserver...

My problem turned out to be bad TLS assets which reside in /etc/kubernetes/ssl/. (In hindsight, it is obvious since it was responding to HTTP requests but not HTTPS.) The log entries in my question were irrelevant to the problem of the TLS assets. The kubelet logs helped me track down the problem:

$ journalctl -u kubelet -e  # jump to end of kubelet logs

The following also came in handy:

$ journalctl -u oem-cloudinit  # if using aws will help you debug your cloud-config
$ journalctl -u etcd2 -f  # follow etcd logs

kube-apiserver refusing external connections on port 443

1 Answers