Kubernetes: Calio not working on remote worker, local ok

Question

I setup a Kubernetes cluster with calico. The setup is "simple"

1x master (local network, ok)
1x node (local network, ok)
1x node (cloud server, not ok)

All debian buster with docker 19.03

On the cloud server the calico pods do not come up:

calico-kube-controllers-token-x:

    Events:
  Type     Reason          Age                   From     Message
  ----     ------          ----                  ----     -------
  Normal   SandboxChanged  47m (x50 over 72m)    kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedMount     43m                   kubelet  MountVolume.SetUp failed for volume "calico-kube-controllers-token-x" : failed to sync secret cache: timed out waiting for the condition
  Normal   SandboxChanged  3m41s (x78 over 43m)  kubelet  Pod sandbox changed, it will be killed and re-created.

calico-node-x:

      Warning  Unhealthy       43m (x5 over 43m)      kubelet  Liveness probe failed: calico/node is not ready: Felix is not live: Get "http://localhost:9099/liveness": dial tcp [::1]:9099: connect: connection refused
  Warning  Unhealthy       14m (x77 over 43m)     kubelet  Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/bird/bird.ctl: connect: no such file or directory
  Warning  BackOff         4m26s (x115 over 39m)  kubelet  Back-off restarting failed container

My guess is that there is something wrong with IP/Network config, but did not figure out which.

Required ports (k8s&BGP) are forwarded from the router, also tried the master directly connected to the internet
--control-plane-endpoint is a hostname and public resolveable
Calico is using BGP peering (using public ip as peer)

This entry does worry me the most:

displayes local ip: kubectl get --raw /api

I tried to find a way to change this to the public IP of the master, without success.

Anyone got a clue what to try next?

Yakuraku Yakuraku · Accepted Answer · 2021-02-19T19:14:49

After an additional time spend with analysis the problem happend to be the distributed api ip address was the local one, not the dns-name.

Created a vpn with wireguard from the cloud node to the local master, so the local ip of the master is reachable from the cloud node.

Don't know if that is the cleanest solution, but it works.

Kubernetes: Calio not working on remote worker, local ok

1 Answers