Using Kubernetes and Weave CNI with multiple clusters in the same subnets

1
votes

I have a situation where I would like to run two kubernetes clusters in the same AWS VPC sharing subnets. This seems to work okay except the weave CNI plugin seems to discover nodes in the other cluster. These nodes get rejected with "IP allocation was seeded by different peers" which makes sense. They are different clusters. Is there a way to keep weave from finding machines in alternate clusters. When I do weave --local status ipam and weave --local status targets I see the expected targets and ipams for each cluster.

Weave pods are in an infinite loop of connecting and rejecting nodes from alternate clusters. This is chewing up cpu and impacting the clusters. If I run kube-utils inside a weave pod it returns the correct nodes for each cluster. It seems kubernetes should know what peers are available, can I just have weave use the peers that the cluster knows about.

After further investigation I believe the issue is that I have scaled machines up and down for both clusters. IP addresses were re-used from one cluster to the next in the process. For instance Cluster A scaled down a node. Weave continues to attempt connections to the now lost node. Cluster B scales up and uses the ip that was used originally in Cluster A. Weave finds the node. This then made weave "discover" the other cluster nodes. Once it discovers one node from the other cluster, it discovers all the nodes.

I have upgraded from 2.4.0 to 2.4.1 to see if some fixes related to re-using ips mitigates this issue.

2
kubernetesweave

2 Answers

1
votes

There is a demo here where Weave Net is run across multiple clusters. This demo was shown in the keynote for KubeCon 2016.

The most important part is here which stops subsequent clusters from forming their own cluster and hence rejecting connections from others.

--ipalloc-init=observer

It's not a particularly clean solution, hacking around with the config, but it does work.

0
votes

I've not tested it, but it seems that option known-peers should do the trick. Weave host should not try to connect to other hosts when connections to all known hosts is established.

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&known-peers=comma-separated list of hosts. Default: empty."

You can find more information about available options for Weave plugin at the section "Changing Configuration Options":