How to add private DNS zone to kube-dns in GKE

Question

I have created a k8 cluster in GKE.

I have a docker registry created in Artifactory, this artifactory is hosted on AWS. I have a route53 entry for docker-repo.aws.abc.com in aws.abc.com Hosted zone in AWS

Now, I need to configure my cluster so that the docker images are pulled from artifactory.

I went through documentation and understand I will have to add stubDomain in my kube-dns configmaps.

kubectl edit cm kube-dns -n kube-system
apiVersion: v1
data:
  stubDomains: |
    {"aws.abc.com" : ["XX.XX.XX.XX"]}
kind: ConfigMap
metadata:
  creationTimestamp: 2019-05-21T14:30:15Z
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kube-dns
  namespace: kube-system
  resourceVersion: "7669"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-dns
  uid: f378aa5f-7bd4-11e9-9df2-42010aa93d03

However, still docker pull command fails.

docker pull docker-repo.aws.abc.com/abc-sampleapp-java/abc-service:V-57bc9c9-201

Error response from daemon: Get https://docker-repo.aws.abc.com/v2/: dial tcp: lookup docker-dev-repo.aws.abc.com on 169.254.169.254:53: no such host

Note: When I make an entry in /etc/hosts file on worker nodes, docker pull works fine.

Sunil Gajula Sunil Gajula · Accepted Answer · 2019-05-31T09:51:28

Pulling an image from registry on pod start uses different DNS settings than when we call DNS from pods inside a cluster.

When Kubernetes starts new pod, it schedules it to the node and then agent on the node named kubelet calls container engine (Docker) to download an image and run it with designed configuration.

Docker uses system DNS to resolve the address of a registry, because it works right on our host system, not in the Kubernetes, that is why any DNS settings will not affect DNS resolving on the image downloading stage. https://github.com/kubernetes/kubernetes/issues/8735 is a discussion about it on Github.

If we want to change DNS settings and override the registry IP to use it on image downloading stage, we should set it in the host system. In the configuration, we need to modify DNS settings on all your nodes in the cluster. The simplest way to do it is using /etc/hosts file and adding a record with your custom IP, e.g. 192.168.1.124 example.com.

After that modifications, Docker on nodes will use the record from /etc/hosts for your registry instead of global DNS records, because that file has higher priority and you will be able to run pods with your image.

To update the host file. you can use a DeamonSet with Security Context as privileged see below:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: hosts-fix-script
  namespace: kube-system
  labels:
    app: hosts-fix-script
spec:
  selector:
    matchLabels:
      name: hosts-fix
  template:
    metadata:
      labels:
        name: hosts-fix
    spec:
      hostPID: true
      containers:
      - name: hosts-fix-script
        image: gcr.io/google-containers/startup-script:v1
        imagePullPolicy: Always
        securityContext:
          privileged: true
        env:
        - name: STARTUP_SCRIPT
          value: |
            #!/bin/bash
                echo "10.0.0.11 onprem.registry" >> /etc/hosts
            echo 'Done'

you need to run the kubectl apply -f

How to add private DNS zone to kube-dns in GKE

1 Answers