Install Kubernetes + Cilium on different networks

Question

There is the following topology:

'left-1', 'left-2', 'right-1', 'right-2', 'center' - hosts (DNS names are same).

"Clouds" - networks.

kubeadm, kubectl, kubelet, docker on all hosts installed correctly.

Kubernetes need install like: 'Master-1' on host 'left-1', 'Master-2' on host 'right-1', and workers on hosts 'left-2' and 'right-2'

All hosts ping each other by the domain name. All ports on all hosts are open. No firewall anywhere.

All hosts have access to the internet.

Here there is a manual to install Kubernetes:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

If I install Kubernetes only on 'left-1' and 'left-2' - all works fine.

If I install Kubernetes only on 'right-1' and 'right-2' - all works fine.

But if I install on all nodes - pods from the left do not connect to pods from right, and right pods do not connect to left pods.

How to install Kubernetes on the left and right nodes together?

I use a Cilium network.

I installed a Cilium network with the command:

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.6.8/install/kubernetes/quick-install.yaml

When i init the first master node, i describe CIDR: 10.217.0.0/16

I tried to install etcd separately from kubernetes. i've got error:

2020-06-25 02:49:37.073290 I | embed: rejected connection from "10.7.0.1:48422" (error "tls: \"10.7.0.1\" does not match any of DNSNames [\"right-1\" \"localhost\"]", ServerName "", IPAddresses ["10.8.1.1" "127.0.0.1" "::1" "10.8.1.1"], DNSNames ["right-1" "localhost"])

10.7.0.1 - it is center, and center is not a part of the etcd cluster. Why etcd checks it?

[left-1]$ traceroute right-1
traceroute to right-1 (10.8.1.1), 30 hops max, 60 byte packets
 1  center (10.7.0.1)  1.381 ms  1.252 ms  1.159 ms
 2  right-1 (10.8.1.1)  1.068 ms  0.990 ms  0.912 ms

I install Cilium with the command: kubectl apply -f raw.githubusercontent.com/cilium/cilium/v1.6.8/install/… — Евгений Колпаков
Is Cilium running on all your Kubernetes nodes when you are using the nodes on the left and right? That's basically your overlay. — Rico
cilium/operator - runs on one node, cilium/cilium runs on all nodes (DaemonSet) — Евгений Колпаков
I forget: i described CIDR: 10.217.0.0/16 when init the first master node — Евгений Колпаков

Евгений Колпаков Евгений Колпаков · Accepted Answer · 2020-06-26T05:58:06

We solved the problem.

Cluster must be created by command:

kubeadm init --config=kubeadm-config.yaml --upload-certs

Where kubeadm-config.yaml contains:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.18.3
controlPlaneEndpoint: "10.7.1.1:6443"
networking:
  podSubnet: "10.217.0.0/16"
etcd:
  local:
    serverCertSANs: ["10.7.1.1", "10.7.2.2", "10.7.0.1", "10.8.1.1", "10.8.2.2", "10.8.0.1"]
    peerCertSANs:   ["10.7.1.1", "10.7.2.2", "10.7.0.1", "10.8.1.1", "10.8.2.2", "10.8.0.1"]

Pay attention to yaml-parameters: serverCertSANs, and peerCertSANs : its contain 10.7.0.1 and 10.8.0.1 - these IPs come to node in network packages as client IP and must be registered as trusted IP. If you have another IPs in inter-node interaction, it must be registered too.

Install Kubernetes + Cilium on different networks

1 Answers