Kubernetes (kubeadm) missing public certificates in containers for outgoing connections

Question

I've setup a kubernetes cluster on my contabo (provider) server(s). Everything is running great so far, till I wanted to access an external SSL domain from a container.

I was able to reproduce the steps by

Setting up a new server
Running kubeadm init on it
Run kubectl run -i --tty test --image=tutum/curl --restart=Never -- sh
Run (in the container)

root@test:/# curl https://acme-v01.api.letsencrypt.org/directory -v
- Hostname was NOT found in DNS cache
- Trying 91.194.91.220...
- Connected to acme-v01.api.letsencrypt.org (91.194.91.220) port 443 (#0)
- successfully set certificate verify locations:
- CAfile: none CApath: /etc/ssl/certs
- SSLv3, TLS handshake, Client hello (1):
- SSLv3, TLS handshake, Server hello (2):
- SSLv3, TLS handshake, CERT (11):
- SSLv3, TLS handshake, Server key exchange (12):
- SSLv3, TLS handshake, Server finished (14):
- SSLv3, TLS handshake, Client key exchange (16):
- SSLv3, TLS change cipher, Client hello (1):
- SSLv3, TLS handshake, Finished (20):
- SSLv3, TLS change cipher, Client hello (1):
- SSLv3, TLS handshake, Finished (20):
- SSL connection using ECDHE-RSA-AES256-GCM-SHA384
- Server certificate:
- subject: CN=contabo.at
- start date: 2017-06-13 03:21:00 GMT
- expire date: 2017-09-11 03:21:00 GMT
- subjectAltName does not match acme-v01.api.letsencrypt.org
- SSL: no alternative certificate subject name matches target host name 'acme-v01.api.letsencrypt.org'
- Closing connection 0
- SSLv3, TLS alert, Client hello (1): curl: (51) SSL: no alternative certificate subject name matches target host name 'acme-v01.api.letsencrypt.org'

If I am running the same thing in a plain docker container like docker run -it --rm tutum/curl /bin/bash (also on the server) everything runs fine.

So this must be a problem with kubernetes.

Can someone help me to add the public SSL certificates to my kubernetes setup?

Thank you & regards Dominik

// EDIT 1

After some further investigations I edited my /etc/resolv.conf that it only contains the Google nameservers.

But this is not the issue.

Everything works until I join another node on my network like: kubeadm join --token db8341.36b4c997b2681683 1.2.3.4:6443

Then it stops working. I am using the weave network / same problem with calico.

// EDIT 2

I added a full code example from my tests:

 root@sh0k:~# kubectl get nodes -o wide
    NAME         STATUS    AGE       VERSION   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION
    c1.sh0k.de   Ready     1m        v1.7.1    <none>        Ubuntu 16.04.2 LTS   4.4.0-83-generic
    sh0k.de      Ready     9m        v1.7.1    <none>        Ubuntu 16.04.2 LTS   4.4.0-83-generic
    root@sh0k:~# kubectl get pods --all-namespaces -o wide
    NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
    kube-system   etcd-sh0k.de                      1/1       Running   1          8m        5.189.140.74    sh0k.de
    kube-system   kube-apiserver-sh0k.de            1/1       Running   1          8m        5.189.140.74    sh0k.de
    kube-system   kube-controller-manager-sh0k.de   1/1       Running   1          8m        5.189.140.74    sh0k.de
    kube-system   kube-dns-2425271678-n3cgv         3/3       Running   3          9m        10.40.0.22      sh0k.de
    kube-system   kube-proxy-fw41z                  1/1       Running   0          1m        213.136.88.53   c1.sh0k.de
    kube-system   kube-proxy-wtd7l                  1/1       Running   1          9m        5.189.140.74    sh0k.de
    kube-system   kube-scheduler-sh0k.de            1/1       Running   1          8m        5.189.140.74    sh0k.de
    kube-system   weave-net-8tts6                   2/2       Running   1          1m        213.136.88.53   c1.sh0k.de
    kube-system   weave-net-smfpn                   2/2       Running   3          5m        5.189.140.74    sh0k.de
    (reverse-i-search)`': ^C
    root@sh0k:~# kubectl run -i --tty test --image=tutum/curl --restart=Never -- sh
    If you don't see a command prompt, try pressing enter.

    # curl -v https://www.google.com
    * Rebuilt URL to: https://www.google.com/
    * Hostname was NOT found in DNS cache
    * Could not resolve host: www.google.com
    * Closing connection 0
    curl: (6) Could not resolve host: www.google.com
    # exit
    root@sh0k:~# kubectl -n kube-system logs kube-dns-2425271678-n3cgv kubedns   
    I0719 05:36:25.156505       7 dns.go:48] version: 1.14.3-4-gee838f6
    I0719 05:36:25.174462       7 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
    I0719 05:36:25.175247       7 server.go:113] FLAG: --alsologtostderr="false"
    I0719 05:36:25.175810       7 server.go:113] FLAG: --config-dir="/kube-dns-config"
    I0719 05:36:25.176361       7 server.go:113] FLAG: --config-map=""
    I0719 05:36:25.176918       7 server.go:113] FLAG: --config-map-namespace="kube-system"
    I0719 05:36:25.177459       7 server.go:113] FLAG: --config-period="10s"
    I0719 05:36:25.177509       7 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
    I0719 05:36:25.177555       7 server.go:113] FLAG: --dns-port="10053"
    I0719 05:36:25.177593       7 server.go:113] FLAG: --domain="cluster.local."
    I0719 05:36:25.177643       7 server.go:113] FLAG: --federations=""
    I0719 05:36:25.177674       7 server.go:113] FLAG: --healthz-port="8081"
    I0719 05:36:25.177702       7 server.go:113] FLAG: --initial-sync-timeout="1m0s"
    I0719 05:36:25.177748       7 server.go:113] FLAG: --kube-master-url=""
    I0719 05:36:25.177779       7 server.go:113] FLAG: --kubecfg-file=""
    I0719 05:36:25.177806       7 server.go:113] FLAG: --log-backtrace-at=":0"
    I0719 05:36:25.177858       7 server.go:113] FLAG: --log-dir=""
    I0719 05:36:25.177887       7 server.go:113] FLAG: --log-flush-frequency="5s"
    I0719 05:36:25.177933       7 server.go:113] FLAG: --logtostderr="true"
    I0719 05:36:25.177961       7 server.go:113] FLAG: --nameservers=""
    I0719 05:36:25.177988       7 server.go:113] FLAG: --stderrthreshold="2"
    I0719 05:36:25.178017       7 server.go:113] FLAG: --v="2"
    I0719 05:36:25.178057       7 server.go:113] FLAG: --version="false"
    I0719 05:36:25.178086       7 server.go:113] FLAG: --vmodule=""
    I0719 05:36:25.178447       7 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
    I0719 05:36:25.179197       7 server.go:198] Skydns metrics enabled (/metrics:10055)
    I0719 05:36:25.179209       7 dns.go:147] Starting endpointsController
    I0719 05:36:25.179215       7 dns.go:150] Starting serviceController
    I0719 05:36:25.179327       7 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
    I0719 05:36:25.179347       7 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
    I0719 05:36:25.679413       7 dns.go:171] Initialized services and endpoints from apiserver
    I0719 05:36:25.679440       7 server.go:129] Setting up Healthz Handler (/readiness)
    I0719 05:36:25.679451       7 server.go:134] Setting up cache handler (/cache)
    I0719 05:36:25.679460       7 server.go:120] Status HTTP port 8081

root@sh0k:~# kubectl -n kube-system logs kube-dns-2425271678-n3cgv dnsmasq
I0719 05:36:25.672456      20 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0719 05:36:25.672682      20 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0719 05:36:25.889812      20 nanny.go:111] 
W0719 05:36:25.889917      20 nanny.go:112] Got EOF from stdout
I0719 05:36:25.890205      20 nanny.go:108] dnsmasq[51]: started, version 2.76 cachesize 1000
I0719 05:36:25.890251      20 nanny.go:108] dnsmasq[51]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0719 05:36:25.890274      20 nanny.go:108] dnsmasq[51]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0719 05:36:25.890287      20 nanny.go:108] dnsmasq[51]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0719 05:36:25.890294      20 nanny.go:108] dnsmasq[51]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0719 05:36:25.890687      20 nanny.go:108] dnsmasq[51]: reading /etc/resolv.conf
I0719 05:36:25.890706      20 nanny.go:108] dnsmasq[51]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0719 05:36:25.890715      20 nanny.go:108] dnsmasq[51]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0719 05:36:25.890722      20 nanny.go:108] dnsmasq[51]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0719 05:36:25.890730      20 nanny.go:108] dnsmasq[51]: using nameserver 8.8.8.8#53
I0719 05:36:25.890740      20 nanny.go:108] dnsmasq[51]: using nameserver 8.8.4.4#53
I0719 05:36:25.891436      20 nanny.go:108] dnsmasq[51]: read /etc/hosts - 7 addresses

I think this is not about the IP address but the cert returned when you curl https://acme-v01.api.letsencrypt.org/directory. subject: CN=contabo.at should be subject: CN=*.api.letsencrypt.org. Can you also paste the curl -v results of your docker command? — Eugene Chow

sfgroups sfgroups · Accepted Answer · 2017-07-17T17:21:03

You're not getting the correct IP address for acme-v01.api.letsencrypt.org DNS hostname.

Can you update your /etc/resolv.conf with external DNS server's IP and test the curl.

eg. Google's DNS server.

nameserver 8.8.8.8

Kubernetes (kubeadm) missing public certificates in containers for outgoing connections

2 Answers