Kubernetes Ingress Controller Fake Certificate on GKE for WSO2 API Management

Question

I encourage an issue regarding setup the TLS Cert-Manager Controller on GKE for WSO2 API Management.

---

I am using WSO2 product Docker images available from WSO2 Private Docker Registry, following the Helm Chart for the deployment of WSO2 API Manager with WSO2 API Manager Analytics on Github (README). And I successfully deployed the WSO2 API Manager with Nginx Ingress Controller (deploy-a-nginx-ingress-and-a-certitificate-manager-controller-on-gke).

---

I want to create a Kubernetes cluster on Google Cloud Platform using an Nginx Ingress Controller to integrate with a certificate manager to automate the process of issue and renew the required certificates.

---

I easily replicate the TLS Cert-Manager Controller on GKE for HelloWorld example from the same medium tutorial (deploy-a-nginx-ingress-and-a-certitificate-manager-controller-on-gke).

hello-app-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-production
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"cert-manager.io/issuer":"letsencrypt-production","kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"true"},"name":"hello-app-ingress","namespace":"default"},"spec":{"rules":[{"host":"test.japangly.xyz","http":{"paths":[{"backend":{"serviceName":"hello-app","servicePort":8080},"path":"/helloworld"}]}}],"tls":[{"hosts":["test.japangly.xyz"],"secretName":"test-japangly-xyz-tls"}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  creationTimestamp: "2020-08-30T04:27:12Z"
  generation: 3
  name: hello-app-ingress
  namespace: default
  resourceVersion: "6478"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/hello-app-ingress
  uid: ea2d8b13-e9b6-4cb0-873d-76ed40253e4f
spec:
  rules:
  - host: test.japangly.xyz
    http:
      paths:
      - backend:
          serviceName: hello-app
          servicePort: 8080
        path: /helloworld
  tls:
  - hosts:
    - test.japangly.xyz
    secretName: test-japangly-xyz-tls
status:
  loadBalancer:
    ingress:
    - ip: 35.239.145.46

However, not working the WSO2 API Management, all I get is

Kubernetes Ingress Controller Fake Certificate

wso2am-pattern-1-am-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-production
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: wso2am-pattern-1
    meta.helm.sh/release-namespace: wso2-apim
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: route
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  creationTimestamp: "2020-08-30T04:41:10Z"
  generation: 4
  labels:
    app.kubernetes.io/managed-by: Helm
  name: wso2am-pattern-1-am-ingress
  namespace: wso2-apim
  resourceVersion: "88840"
  selfLink: /apis/extensions/v1beta1/namespaces/wso2-apim/ingresses/wso2am-pattern-1-am-ingress
  uid: 58f4b549-a565-493b-9f9f-72ad76877819
spec:
  rules:
  - host: am.japangly.xyz
    http:
      paths:
      - backend:
          serviceName: wso2am-pattern-1-am-service
          servicePort: 9443
        path: /
  tls:
  - hosts:
    - am.japangly.xyz
    secretName: am-japangly-xyz-tls
status:
  loadBalancer:
    ingress:
    - ip: 35.239.145.46

Answer 1

As I said in the comment:

I ran setup like yours and noticed that the cert-manager was creating the secret but was not provisioning it further. Issuer is namespaced resource and needs to be in namespace where your Ingress resides. Please tell if your namespace wso2-apim have the Issuer needed to provide the certificate. For troubleshooting you can run $ kubectl describe certificate -n namespace. Also the fake Kubernetes certificate is used when there is an issue with a tls: secret part.

I wanted to give more insight on what the potential issue may be and some other tips working with nginx-ingress.

---

Certificate showing as Kubernetes Ingress Controller Fake Certificate will kick in when there are issues with the actual secret storing the certificate used in the Ingress definition.

One of the possible situations where Fake Certificate will kick in is in the lack of the actual secret with a certificate.

---

As pointed in the part of my comment, Issuer is a namespaced resource and it needs to be in a namespace that Ingress and secret is created. It will create a secret but it will not progress further with signing.

Looking on your setup:

• nginx-ingress controller spawned in nginx namespace (GOOD)
• cert-manager spawned in namespace cert-manager namespace (GOOD)
• Issuer for certificates spawned in default namespace (POTENTIAL ISSUE)
• WSO2 application spawned in wso2-apim namespace (POTENTIAL ISSUE)

To make it work you can either:

• Run your WSO2 application in default namespace same as Issuer
• Create an Issuer in wso2-apim namespace
• Create a ClusterIssuer

As pointed by official documentation:

An Issuer is a namespaced resource, and it is not possible to issue certificates from an Issuer in a different namespace. This means you will need to create an Issuer in each namespace you wish to obtain Certificates in.

Cert-manager.io: Docs: Concepts: Issuer

---

As for troubleshooting steps you can invoke following commands:

• $ kubectl describe issuer ISSUER_NAME -n namespace
• $ kubectl describe certificate CERTIFICATE_NAME -n namespace
• $ kubectl describe secret SECRET_NAME -n namespace

---

Assuming that:

• You have a working Kubernetes cluster
• You have spawned nginx-ingress and have the LoadBalancer IP provisioned correctly
• You have a domain name pointing to the LoadBalancer IP of your nginx-ingress controller

After that you spawned:

• example namespace
• hello-app as in medium guide:
  • $ kubectl create deployment hello-app --image=gcr.io/google-samples/hello-app:1.0 -n example
• exposed it locally
  • $ kubectl expose deployment hello-app --type=NodePort --port=8080 -n example
• created an Ingress resource like below:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: example
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - super.example.com
    secretName: super-example-tls
  rules:
  - host: super.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: hello-app
          servicePort: 8080

With no Issuer in the example namespace the logs from the certificate will look like that:

• $ kubectl describe certificate super-example-tls -n example

Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Issuing    6m33s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  6m33s  cert-manager  Stored new private key in temporary Secret resource "super-example-tls-XXXXX"
  Normal  Requested  6m33s  cert-manager  Created new CertificateRequest resource "super-example-tls-XXXXX"

Issuer.yaml used in an example for more reference:

kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: example
spec: 
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: PUT_EMAIL_ADDRESS_HERE
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

After you create an Issuer you should see a new event in certificate:

  Normal  Issuing    25s    cert-manager  The certificate has been successfully issued