Guestbook example not working. Unable to connect to MASTER

Question

I have four minions and one master on ubuntu cluster. And I'm following guestbook example. Redis slave is not communicating with master.

# kubectl get nodes

NAME          LABELS                                       STATUS
172.16.2.73   kubernetes.io/hostname=172.16.2.73           Ready
172.16.2.91   kubernetes.io/hostname=172.16.2.91           Ready
172.16.2.92   kubernetes.io/hostname=172.16.2.92           Ready
172.16.2.96   kubernetes.io/hostname=172.16.2.96           Ready

# kubectl get pods

POD                  IP           CONTAINER(S)   IMAGE(S)                                         HOST                      LABELS                                                              STATUS    CREATED         MESSAGE
kube-dns-99zoq       10.1.95.2                                                                    172.16.2.91/172.16.2.91   k8s-app=kube-dns,kubernetes.io/cluster-service=true,name=kube-dns   Running   About an hour
                                  kube2sky       gcr.io/google_containers/kube2sky:1.1                                                                                                          Running   About an hour   last termination: exit code 1
                                  skydns         gcr.io/google_containers/skydns:2015-03-11-001                                                                                                 Running   About an hour
                                  etcd           quay.io/coreos/etcd:v2.0.3                                                                                                                     Running   About an hour
redis-master-eq6ig   10.1.39.16                                                                   172.16.2.96/172.16.2.96   name=redis-master                                                   Running   About an hour
                                  master         redis                                                                                                                                          Running   About an hour
redis-slave-f394k    10.1.34.2                                                                    172.16.2.73/172.16.2.73   name=redis-slave                                                    Running   About an hour
                                  slave          kubernetes/redis-slave:v2                                                                                                                      Running   About an hour
redis-slave-t690b    10.1.46.2                                                                    172.16.2.92/172.16.2.92   name=redis-slave                                                    Running   About an hour
                                  slave          kubernetes/redis-slave:v2                                                                                                                      Running   About an hour

# kubectl get services

NAME            LABELS                                                              SELECTOR            IP(S)        PORT(S)
kube-dns        k8s-app=kube-dns,kubernetes.io/cluster-service=true,name=kube-dns   k8s-app=kube-dns    15.1.3.10    53/UDP
kubernetes      component=apiserver,provider=kubernetes                             <none>              15.1.3.2     443/TCP
kubernetes-ro   component=apiserver,provider=kubernetes                             <none>              15.1.3.1     80/TCP
redis-master    name=redis-master                                                   name=redis-master   15.1.3.153   6379/TCP
redis-slave     name=redis-slave                                                    name=redis-slave    15.1.3.181   6379/TCP

# kubectl logs redis-slave-t690b

[9] 29 Jun 07:19:15.432 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
[9] 29 Jun 07:19:15.432 * The server is now ready to accept connections on port 6379
[9] 29 Jun 07:19:16.432 * Connecting to MASTER redis-master:6379
[9] 29 Jun 07:19:31.581 # Unable to connect to MASTER: No such file or directory
[9] 29 Jun 07:19:32.584 * Connecting to MASTER redis-master:6379
[9] 29 Jun 07:19:47.743 # Unable to connect to MASTER: No such file or directory
[9] 29 Jun 07:19:48.746 * Connecting to MASTER redis-master:6379

# cat /var/log/upstart/kube-proxy.log

E0629 00:12:00.166619   23844 api.go:180] Unable to load endpoints: Get http://172.16.2.73:8080/api/v1beta3/endpoints: dial tcp 172.16.2.73:8080: connection refused
E0629 00:12:00.171629   23844 api.go:108] Unable to load services: Get http://172.16.2.73:8080/api/v1beta3/services: dial tcp 172.16.2.73:8080: connection refused
E0629 00:22:15.168869   23844 proxier.go:153] Failed to ensure iptables: error creating chain "KUBE-PORTALS-CONTAINER": exit status 4: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
E0629 00:22:15.180001   23844 proxier.go:391] Failed to install iptables KUBE-PORTALS-CONTAINER rule for service "default/redis-master:"
E0629 00:22:15.180121   23844 proxier.go:168] Failed to ensure portal for "default/redis-master:": error checking rule: exit status 4: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

# cat /var/log/upstart/flanneld.log

E0629 00:12:19.331286 23841 main.go:171] Failed to create SubnetManager: 100: Key not found (/coreos.com) [27]
I0629 00:12:20.333500 23841 main.go:278] Installing signal handlers
I0629 00:12:20.333742 23841 main.go:138] Determining IP address of default interface
I0629 00:12:20.334787 23841 main.go:228] Using 172.16.2.73 as external interface
I0629 00:12:20.336418 23841 subnet.go:334] Picking subnet in range 10.1.1.0 ... 10.1.255.0
I0629 00:12:20.900225 23841 subnet.go:97] Subnet lease acquired: 10.1.95.0/24
I0629 00:12:20.907356 23841 main.go:245] UDP mode initialized
I0629 00:12:20.907526 23841 udp.go:212] Watching for new subnet leases
I0629 00:12:20.909248 23841 udp.go:237] Subnet added: 10.1.74.0/24
I0629 00:12:21.340083 23841 udp.go:237] Subnet added: 10.1.8.0/24
I0629 00:12:21.340703 23841 udp.go:237] Subnet added: 10.1.96.0/24

Looks like everything is working fine except proxy. How to configure proxy, I didn't find any proxy things in guestbook example?

How did you start your cluster, and what version of Kubernetes are you running? — lavalamp

priyanka_rao priyanka_rao · Accepted Answer · 2015-07-29T11:39:24

I faced similar issue this link helped : https://github.com/GoogleCloudPlatform/kubernetes/issues/6158 see hack solution by kcao3.

Following are the changes :

add redis-master IP in slaves container /etc/hosts file
add redis-slave IP and redis-master IP in frontend containers in /etc/hosts file

Note: also check firewall settings, if you are using aws cloud, make sure security group have needed ports enabled.

Guestbook example not working. Unable to connect to MASTER

1 Answers