kubeadm init phase upload-config failing

Question

I am new to kubernetes and want to setup kubernetes HA setup after successfully completing examples with minikube and single master kubernetes cluster. I am using AWS EC2 instances and AWS application load balancer for this purpose. I dont want to use KOPS or any other tool for installation. I want to get hands on with kubeadm. I followed below steps

1. Created self signed certificate ca.crt and ca.key to use for kubernetes
2. Installed this certificate as Root CA on my ubuntu instance
3. Copied this ca.crt and ca.key to /etc/kubernetes/pki
4. Created new certificate for aws loadbalancer and signed it with above ca.crt. With this certificate created Aws application load balancer
5. I also created Record Set in AWS Route53 for domain name mapping. I also made sure this domain name mapping is working. (i.e. master.k8sonaws.com is properly resolving to aws load balancer)

6. Now I am using kubeadm init

   kubeadm init --pod-network-cidr=192.168.0.0/20
   --service-cidr=192.168.16.0/20 --node-name=10.0.0.13  --control-plane-endpoint "master.k8sonaws.com:443"  --upload-certs --v=8 --apiserver-bind-port=443 --apiserver-cert-extra-sans=master.k8sonaws.com,i-0836dd4dc6609a924
   

7. This command is succeeding up-to upload-config phase. Health check endpoint is returning success but after that its failing in upload-config phase

   configmaps is forbidden: User "system:anonymous" cannot create resource "configmaps" in API group "" in the namespace "kube-system"

Here I am not able to understand why kubeadm is passing anonymyous user for api call. How can I resolve this issue

Answer 1

The certificate in the kubeconfig file used to create the ConfigMap does not have right groups. I would say don't generate ca and cert for kubernetes yourself. Just use kubeadm init and kubeadm will handle the ca and cert generation. After your kubernetes cluster is up and running you can use the same ca to generate a cert yourself and use that in aws load balancer.

Answer 2

I found solution to this problem after trying lot of different things for 2 days. Problem is aws load balancer does not pass client certificate to backend server when using https listener. And problem is Aws has not documented this fact or I did not get those document if they are there.

Way to solve this problem is using plane http listener using same port 443. So SSL termination is carried out by backened server. In my case this is not security threat since both my load balancer and backend servers are internal and not exposed to public ip.