kube-apiserver is not able to get or set keys when started

2
votes

im trying to setup a a high available kubernetes cluster with packer and terraform instead the kube-up.sh scripts. Reason: I want bigger machines, different setup etc. Most of my configuration comes from the coreos kubernetes deployment tutorial.

Something about my setup:

CoreOS

Everything runs on gce. Ive got 3 etcd and one skydns instances. They are working and able to reach each other.

I have one instance as kubernetes master instance that is running the kubelet with manifests.

My actual problem right now is that the kube-api server is not able to connect to it self. I can run a curl command from my host system with valid response. /version and others.

It is also a little bit strange that 443 and 8080 are not forwarded from docker. Or is this a normal behavior?

I thought i missconfigured some master endpoints. so i tried localhost and the external ip for all manifests. => Not working.

Errors in the kube-api container:

I0925 14:51:47.505859       1 plugins.go:69] No cloud provider specified.
I0925 14:51:47.973450       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
E0925 14:51:48.009367       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010730       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010996       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.011083       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012697       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012753       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] listing is available at https://104.155.60.74:443/swaggerapi/
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] https://104.155.60.74:443/swaggerui/ is mapped to folder /swagger-ui/
I0925 14:51:48.136166       1 server.go:441] Serving securely on 0.0.0.0:443
I0925 14:51:48.136248       1 server.go:483] Serving insecurely on 127.0.0.1:8080

The controller container has nearly the same erros. Every other container is fine.

My config:

/etc/kubelet.env

KUBE_KUBELET_OPTS="\
  --api_servers=http://127.0.0.1:8080 \
  --register-node=false \
  --allow-privileged=true \
  --config=/etc/kubernetes/manifests \
  --tls_cert_file=/etc/kubernetes/ssl/apiserver.pem \
  --tls_private_key_file=/etc/kubernetes/ssl/apiserver-key.pem \
  --cloud-provider=gce \
  --cluster_dns=10.10.38.10 \
  --cluster_domain=cluster.local \
  --cadvisor-port=0"

/etc/kubernetes/manifests/

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    - /hyperkube
    - apiserver
    - --bind-address=0.0.0.0
    - --etcd_servers=http://10.10.125.10:2379,http://10.10.82.201:2379,http://10.10.63.185:2379
    - --allow-privileged=true
    - --service-cluster-ip-range=10.40.0.0/16
    - --secure_port=443
    - --advertise-address=104.155.60.74
    - --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --client-ca-file=/etc/kubernetes/ssl/ca.pem
    - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    ports:
    - containerPort: 443
      hostPort: 443
      name: https
    - containerPort: 8080
      hostPort: 8080
      name: local
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

/etc/kubernetes/manifests/kube-controller-manager.yml

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - name: kube-controller-manager
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    - /hyperkube
    - controller-manager
    - --master=https://104.155.60.74:443
    - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --root-ca-file=/etc/kubernetes/ssl/ca.pem
    - --cloud_provider=gce
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

docker ps

CONTAINER ID        IMAGE                                       COMMAND                CREATED             STATUS              PORTS               NAMES
3e37b2ea2277        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube controll   31 minutes ago      Up 31 minutes                           k8s_kube-controller-manager.afecd3c9_kube-controller-manager-kubernetes-km0.c.stylelounge-1042.inte
rnal_kube-system_621db46bf7b0764eaa46d17dfba8e90f_519cd0da   
43917185d91b        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube proxy --   31 minutes ago      Up 31 minutes                           k8s_kube-proxy.a2db3197_kube-proxy-kubernetes-km0.c.stylelounge-1042.internal_kube-system_67c22e99a
eb1ef9c2997c942cfbe48b9_c82a8a60                             
f548279e90f9        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube apiserve   31 minutes ago      Up 31 minutes                           k8s_kube-apiserver.2bcb2c35_kube-apiserver-kubernetes-km0.c.stylelounge-1042.internal_kube-system_8
67c500deb54965609810fd0771fa92d_a306feae                     
94b1942a09f0        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube schedule   31 minutes ago      Up 31 minutes                           k8s_kube-scheduler.603b59f4_kube-scheduler-kubernetes-km0.c.stylelounge-1042.internal_kube-system_3
9e2c582fd067b44ebe8cefaee036c0e_e0ddf6a2                     
9de4a4264ef6        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-s   31 minutes ago      Up 31 minutes                           k8s_controller-manager-elector.89f472b4_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_k
ube-system_e23fc0902c7e6da7b315ad34130b9807_7c8d2901         
af2df45f4081        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-s   31 minutes ago      Up 31 minutes                           k8s_scheduler-elector.608b6780_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_kube-syste
m_e23fc0902c7e6da7b315ad34130b9807_b11e601d                  
ac0e068456c7        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-controller-manager-kubernetes-km0.c.stylelounge-1042.internal_kube-system_621d
b46bf7b0764eaa46d17dfba8e90f_e9760e28                        
2773ba48d011        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_kube-system_e23fc0902c7e6
da7b315ad34130b9807_4fba9edb                                 
987531f1951d        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-apiserver-kubernetes-km0.c.stylelounge-1042.internal_kube-system_867c500deb549
65609810fd0771fa92d_d15d2d66                                 
f4453b948186        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-proxy-kubernetes-km0.c.stylelounge-1042.internal_kube-system_67c22e99aeb1ef9c2
997c942cfbe48b9_07e540c8                                     
ce01cfda007e        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-scheduler-kubernetes-km0.c.stylelounge-1042.internal_kube-system_39e2c582fd067
b44ebe8cefaee036c0e_e6cb6500

Here the curl command:

kubernetes-km0 ~ # docker logs a404a310b55e
I0928 09:14:05.019135       1 plugins.go:69] No cloud provider specified.
I0928 09:14:05.192451       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
I0928 09:14:05.192900       1 master.go:295] Will report 10.10.247.127 as public IP address.
E0928 09:14:05.226222       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226428       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226479       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226593       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226908       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/28 09:14:05 log.go:30: [restful/swagger] listing is available at https://10.10.247.127:443/swaggerapi/
[restful] 2015/09/28 09:14:05 log.go:30: [restful/swagger] https://10.10.247.127:443/swaggerui/ is mapped to folder /swagger-ui/
E0928 09:14:05.232632       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
I0928 09:14:05.368697       1 server.go:441] Serving securely on 0.0.0.0:443
I0928 09:14:05.368788       1 server.go:483] Serving insecurely on 127.0.0.1:8080
kubernetes-km0 ~ # curl http://127.0.0.1:8080/api/v1/limitranges
{
  "kind": "LimitRangeList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/limitranges",
    "resourceVersion": "100"
  },
  "items": []
}
2
kubernetes

2 Answers

1
votes

You need to register the master as a node if you want the master to actually host any pods with the --register-node=true flag to the kubelet runnning on master. The CoreOs tutorial does not register the master as a node because thats the ideal scenario.

0
votes

I believe you need to specify --insecure-address=127.0.0.1 and --insecure-port=8080 to open up on HTTP, the default is https.