Failed to setup a HA etcd cluster

Question

I want to setup an etcd cluster runnin on multiple nodes. I have running 2 unbuntu 18.04 machines running on a Hyper-V terminal.

I followed this guide on the official kubernetes site: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/

Therefore, I changed the several scripts and executed this scripts on HOST0 and HOST1

export HOST0=192.168.101.90
export HOST1=192.168.101.91

mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/


ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("infra0" "infra1")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${HOST}"
        peerCertSANs:
        - "${HOST}"
        extraArgs:
            initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380
            initial-cluster-state: new
            name: ${NAME}
            listen-peer-urls: https://${HOST}:2380
            listen-client-urls: https://${HOST}:2379
            advertise-client-urls: https://${HOST}:2379
            initial-advertise-peer-urls: https://${HOST}:2380
EOF
done

After that, I executed this command on HOST0

kubeadm init phase certs etcd-ca

I created all the nessecary on HOST0

# cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.k
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete


kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0

# clean up certs that should not be copied off this host
find /tmp/${HOST1} -name ca.key -type f -delete

After that, I copied the files to the second ETCTD node (HOST1). Before that I created a root user mbesystem

USER=mbesystem
 HOST=${HOST1}
 scp -r /tmp/${HOST}/* ${USER}@${HOST}:
 ssh ${USER}@${HOST}
 USER@HOST $ sudo -Es
 root@HOST $ chown -R root:root pki
 root@HOST $ mv pki /etc/kubernetes/

I'll check all the files were there on HOST0 and HOST1.

On HOST0 I started the etcd cluster using:

kubeadm init phase etcd local --config=/tmp/192.168.101.90/kubeadmcfg.yaml

On Host1 I started using:

kubeadm init phase etcd local --config=/home/mbesystem/kubeadmcfg.yaml

After I executed:

docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.4.3-0 etcdctl \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://192.168.101.90:2379 endpoint health --cluster

I discovered my cluster is not healty, I'll received a connection refused.

I can't figure out what went wrong. Any help will be appreciated.

Answer 1

I've looked into it, reproduced what was in the link that you provided: Kubernetes.io: Setup ha etcd with kubeadm and managed to make it work.

Here is some explanation/troubleshooting steps/tips etc.

First of all etcd should be configured with odd number of nodes. What I mean by that is it should be created as 3 or 5 nodes cluster.

Why an odd number of cluster members?

An etcd cluster needs a majority of nodes, a quorum, to agree on updates to the cluster state. For a cluster with n members, quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for quorum. Although adding a node to an odd-sized cluster appears better since there are more machines, the fault tolerance is worse since exactly the same number of nodes may fail without losing quorum but there are more nodes that can fail. If the cluster is in a state where it can't tolerate any more failures, adding a node before removing nodes is dangerous because if the new node fails to register with the cluster (e.g., the address is misconfigured), quorum will be permanently lost.

-- Github.com: etcd documentation

---

Additionally here are some troubleshooting steps:

• Check if docker is running You can check it by running command (on systemd installed os): $ systemctl show --property ActiveState docker
• Check if etcd container is running properly with: $ sudo docker ps
• Check logs of etcd container if it's running with: $ sudo docker logs ID_OF_CONTAINER

---

How I've managed to make it work:

Assuming 2 Ubuntu 18.04 servers with IP addresses of:

• 10.156.0.15 and name: etcd-1
• 10.156.0.16 and name: etcd-2

Additionally:

• SSH keys configured for root access
• DNS resolution working for both of the machines ($ ping etcd-1)

Steps:

• Pre-configuration before the official guide.

  I did all of the below configuration with the usage of root account

• Configure the kubelet to be a service manager for etcd.
• Create configuration files for kubeadm.
• Generate the certificate authority.
• Create certificates for each member
• Copy certificates and kubeadm configs.
• Create the static pod manifests.
• Check the cluster health.

Pre-configuration before the official guide

Pre-configuration of this machines was done with this StackOverflow post with Ansible playbooks: Stackoverflow.com: 3 kubernetes clusters 1 base on local machine

You can also follow official documentation: Kubernetes.io: Install kubeadm

Configure the kubelet to be a service manager for etcd.

Run below commands on etcd-1 and etcd-2 with root account.

cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
#  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
EOF

$ systemctl daemon-reload

$ systemctl restart kubelet

Create configuration files for kubeadm.

Create your configuration file on your etcd-1 node.

Here is modified script that will create kubeadmcfg.yaml for only 2 nodes:

export HOST0=10.156.0.15
export HOST1=10.156.0.16

# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/

ETCDHOSTS=(${HOST0} ${HOST1})
NAMES=("etcd-1" "etcd-2")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${HOST}"
        peerCertSANs:
        - "${HOST}"
        extraArgs:
            initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380
            initial-cluster-state: new
            name: ${NAME}
            listen-peer-urls: https://${HOST}:2380
            listen-client-urls: https://${HOST}:2379
            advertise-client-urls: https://${HOST}:2379
            initial-advertise-peer-urls: https://${HOST}:2380
EOF
done

Take a special look on:

• export HOSTX on the top of the script. Paste the IP addresses of your machines there.
• NAMES=("etcd-1" "etcd-2"). Paste the names of your machines (hostname) there.

Run this script from root account and check if it created files in /tmp/IP_ADDRESS directory.

Generate the certificate authority

Run below command from root account on your etcd-1 node:

$ kubeadm init phase certs etcd-ca

Create certificates for each member

Below is a part of the script which is responsible for creating certificates for each member of etcd cluster. Please modify the HOST0 and HOST1 variables.

#!/bin/bash
HOST0=10.156.0.15
HOST1=10.156.0.16

kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0

Run above script from root account and check if there is pki directory inside /tmp/10.156.0.16/.

There shouldn't be any pki directory inside /tmp/10.156.0.15/ as it's already in place.

Copy certificates and kubeadm configs.

Copy your kubeadmcfg.yaml of etcd-1 from /tmp/10.156.0.15 to root directory with: $ mv /tmp/10.156.0.15/kubeadmcfg.yaml /root/

Copy the content of /tmp/10.156.0.16 from your etcd-1 to your etcd-2 node to /root/ directory:

$ scp -r /tmp/10.156.0.16/* [email protected]:

After that check if files copied correctly, have correct permissions and copy pki folder to /etc/kubernetes/ with command on etcd-2:

$ mv /root/pki /etc/kubernetes/

Create the static pod manifests.

Run below command on etcd-1 and etcd-2:

$ kubeadm init phase etcd local --config=/root/kubeadmcfg.yaml

All should be running now.

Check the cluster health.

Run below command to check cluster health on etcd-1.

docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.4.3-0 etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert /etc/kubernetes/pki/etcd/ca.crt --endpoints https://10.156.0.15:2379 endpoint health --cluster

Modify: --endpoints https://10.156.0.15:2379 with correct IP address of etcd-1

It should give you a message like this:

https://10.156.0.15:2379 is healthy: successfully committed proposal: took = 26.308693ms
https://10.156.0.16:2379 is healthy: successfully committed proposal: took = 26.614373ms

Above message concludes that the etcd is working correctly but please be aware of an even number of nodes.

Please let me know if you have any questions to that.