Why does Google Cloud show an error when using ClusterIP

Question

In my gcloud console it shows the following error for my defined ingresses:

Error during sync: error while evaluating the ingress spec: service "monitoring/kube-prometheus" is type "ClusterIP", expected "NodePort" or "LoadBalancer"

I am using traefik as reverse proxy (instead of nginx) and therefore I define an ingress using a ClusterIP. As far as I understand the process all traffic is proxied through the traefik service (which has a Loadbalancer ingress defined) and therefore all my other ingresses SHOULD actually have a ClusterIP instead of NodePort or Loadbalancer?

Question:

So why does Google Cloud warn me that it expected a NodePort or LoadBalancer?

Answer 1

I don't know why that error happens, because it seems (to me) to be a valid configuration. But to clear the error, you can switch your service to a named NodePort. Then switch your ingress to use the port name instead of the number. For example:

Service:

apiVersion: v1
kind: Service
metadata:
  name: testapp
spec:
  ports:
  - name: testapp-http # ADD THIS
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: testapp
  type: NodePort

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: testapp
spec:
  rules:
  - host: hostname.goes.here
    http:
      paths:
      - backend:
          serviceName: testapp
          # USE THE PORT NAME FROM THE SERVICE INSTEAD OF THE PORT NUMBER
          servicePort: testapp-http
        path: /

Update:

This is the explanation I received from Google.

Since services by default are ClusterIP [1] and this type of service is meant to be accessible from inside the cluster. It can be accessed from outside when kube-proxy is used, not meant to be directly accessed with an ingress.

As a suggestion, I personally find this article [2] good for understanding the difference between these types of services.

[1] https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

[2] https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

Answer 2

Thanks @aayore. In my case, I had to specify an ingress class explicitly, so that Google Cloud wouldn't interfere. The Nginx ingress seems to be happy with ClusterIp services.

metadata:
  name: foo
  annotations:
    kubernetes.io/ingress.class: "nginx"

Answer 3

We were having this issue on our production environment, but not on staging, and it was being caused by a mismatch in cluster versions. It turns out that ClusterIP is only a valid ServiceType on GKE if you're using Container Native Load Balancing, which is enabled by default on GKE clusters with version 1.17.6-gke.7 and up. We fixed it by simply upgrading our production cluster to the latest stable version.

Answer 4

For us the solution was to set the annotation cloud.google.com/neg: '{"ingress": true}' on the services to be exposed.

Usually this is set automatically to all services under the following conditions:

• services created with at least 1.17.6-gke.7
• VPC-native clusters
• Not using a Shared VPC
• Not using GKE Network Policy

As we started to introduce network policies our exposed services stopped working.

So the example above should also work with ClusterIPs:

apiVersion: v1
kind: Service
metadata:
  name: testapp
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
spec:
  ports: ...
  type: ClusterIP