Kubernetes: VPN server and DNS issues

Question

I spinned a docker-openvpn container in my (local) Kubernetes cluster to access my Services securely and debug dependent services locally.

I can connect to the cluster via the openVPN server. However I can't resolve my Services via DNS.

I managed to get to the point where after setting routes on the VPN server:

I can ping a Pod by IP (subnet 10.2.0.0/16)
I can ping a Service by IP (subnet 10.3.0.0/16 like the DNS which is at 10.3.0.10)
I can curl to a Services by IP and get the data I need.

but when i nslookup kubernetes or any Service, I get:

nslookup kubernetes
;; Got recursion not available from 10.3.0.10, trying next server
;; Got SERVFAIL reply from 10.3.0.10, trying next server

I am still missing something for the data to return from the DNS server, but can't figure what I need to do.

How do I debug this SERVFAIL issue in Kubernetes DNS?

EDIT:

Things I have noticed and am looking to understand:

nslookup works to resolve Service name in any pod except the openvpn Pod
while nslookup works in those other Pods, ping does not.
similarly traceroute in those other Pods leads to the flannel layer 10.0.2.2 and then stops there.

from this I guess ICMP must be blocked at the flannel layer, and that doesn't help me figure where DNS is blocked.

EDIT2:

I finally figured how to get nslookup to work: I had to push the DNS search domain to the client with

push "dhcp-option DOMAIN-SEARCH cluster.local"
push "dhcp-option DOMAIN-SEARCH svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH default.svc.cluster.local"

add with the -p option in the docker-openvpn image

so i end up with

docker run -v /etc/openvpn:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig \
-u udp://192.168.10.152:1194 \
-n 10.3.0.10 \
-n 192.168.10.1 \
-n 8.8.8.8 \
-n 75.75.75.75 \
-n 75.75.75.76 \
-s 10.8.0.0/24 \
-d \
-p "route 10.2.0.0 255.255.0.0" \
-p "route 10.3.0.0 255.255.0.0" \
-p "dhcp-option DOMAIN cluster.local" \
-p "dhcp-option DOMAIN-SEARCH svc.cluster.local" \
-p "dhcp-option DOMAIN-SEARCH default.svc.cluster.local"

Now, nslookup works but curl still does not

github.com/kylemanna/docker-openvpn describes creating configuration files and certificates in a docker-volume. How did you created configuration files and certificates, which would be valid for openvpn server running in a k8s-cluster? Then how did you mount those files in k8s-container? Using secrets? — Manojkumar Khotele
sorry i never replied to this comment: I opened a PR to do this for k8s github.com/kylemanna/docker-openvpn/pull/98 with all you need. This is old, and there is now other options like github.com/redspread/kube-openvpn — MrE

MrE MrE · Accepted Answer · 2016-01-20T16:55:23

finally my config looks like this:

docker run -v /etc/openvpn:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig \
-u udp://192.168.10.152:1194 \
-n 10.3.0.10 \
-n 192.168.10.1 \
-n 8.8.8.8 \
-n 75.75.75.75 \
-n 75.75.75.76 \
-s 10.8.0.0/24 \
-N \
-p "route 10.2.0.0 255.255.0.0" \
-p "route 10.3.0.0 255.255.0.0" \
-p "dhcp-option DOMAIN-SEARCH cluster.local" \
-p "dhcp-option DOMAIN-SEARCH svc.cluster.local" \
-p "dhcp-option DOMAIN-SEARCH default.svc.cluster.local"

-u for the VPN server address and port

-n for all the DNS servers to use

-s to define the VPN subnet (as it defaults to 10.2.0.0 which is used by Kubernetes already)

-d to disable NAT

-p to push options to the client

-N to enable NAT: it seems critical for this setup on Kubernetes

the last part, pushing the search domains to the client, was the key to getting nslookup etc.. to work.

note that curl didn't work at first, but seems to start working after a few seconds. So it does work but it takes a bit of time for curl to be able to resolve.

Kubernetes: VPN server and DNS issues

2 Answers