nginx ingress w/ gke tcp loadbalancer and TLS certificate

Question

I'm running a cluster on GKE with the nginx-ingress controller instead of using the default gce ingress controller.

I'm generating certificates using cert-manager. If I use the default gce controller, this works fine.

I've deployed the nginx-ingress controller with the following config:

containers:
  - args:
    - /nginx-ingress-controller
    - --default-backend-service=kube-system/nginx-ingress-default-backend
    - --publish-service=kube-system/nginx-ingress-controller
    - --election-id=ingress-controller-leader
    - --ingress-class=nginx
    - --configmap=kube-system/nginx-ingress-controller
    - --sort-backends=true

The service for this uses the default service type=LoadBalancer and I can see in the Google Cloud console that I have been provisioned a TCP layer 4 loadbalancer.

My frontend service and ingress has the following annotations:

annotations:
  ingress.kubernetes.io/allow-http: "true"
  kubernetes.io/ingress.class: nginx
  ingress.kubernetes.io/ssl-redirect: "true"
  kubernetes.io/tls-acme: "true"

HTTP works without an issue, but when I try over HTTPS I get an SSL protocol error:

* Rebuilt URL to: https://hostname/
*   Trying xx.xxx.xx.xxx...
* TCP_NODELAY set
* Connected to guestbook.lbrlabs.com (xx.xxx.xx.xxx) port 443 (#0)
* Unknown SSL protocol error in connection to hostname:-9847
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to hostname:-9847

and I can see what looks to be binary data in the logs:

10.36.0.1 - [10.36.0.1] - - [04/Aug/2018:23:37:23 +0000] "\x16\x03\x01\x00\xC8\x01\x00\x00\xC4\x03\x03[f8\xB4J\xFD\xB6\x1C9\x93\xA9\xF6$\x82\x1B~h" 400 174 "-" "-" 0 0.051 [] - - - -

What configuration option am I missing here? I tried using proxy-protocol but that didn't work (as I expected). I'm using 0.14.0 of the nginx-ingress-controller.

coolinuxoid coolinuxoid · Accepted Answer · 2019-10-27T12:25:17

It would be helpful if you put your whole ingress yaml to the post, but from first view if you create your TLS certificate via Ingress, then I think you miss issuer in annotations, because cert-manager checks this annotation to issue the certificate. Check below:

kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/issuer: letsencrypt-prod
    ...
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: backend-service
          servicePort: http
        path: /
  tls:
  - hosts:
    - example.com
    secretName: example-com-tls

You can describe your ingress to see the events as well:

kubectl describe ingress <ingress-name>

If this is not the case, then you have to troubleshoot your certificate if everything ok with them, via:

kubectl get certs # To see the status of certificates
kubectl describe cert # If it is issued successfully

Also you can check logs of cert-manager controller how the certificate is issued.

kubectl logs cert-manager-pod-name-21322 -n cert-manager

Hope it helps!

nginx ingress w/ gke tcp loadbalancer and TLS certificate

1 Answers