Kubernetes cert-manager not updating certificates after issuer change

Question

I am using cert-manager 0.5.2 to manage Let's Encrypt certificates on our Kubernetes cluster.

I was using the Let's Encrypt staging environment, but have now moved to use their production certificates. The problem is that my applications aren't updating to the new, valid certificates.

I must have screwed something up while updating the issuer, certificate, and ingress resources, but I can't see what. I have also reinstalled the NGINX ingress controller and cert-manager, and recreated my applications, but I am still getting old certificates. What can I do next?

Describing the letsencrypt cluster issuer:

Name:         letsencrypt
Namespace:
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt","namespace":""},"spec":{"acme":{"e...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Cluster Name:
  Creation Timestamp:  2019-01-04T09:27:49Z
  Generation:          0
  Resource Version:    130088
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/letsencrypt
  UID:                 00f0ea0f-1003-11e9-997f-ssh3b4bcc625
Spec:
  Acme:
    Email:  [email protected]
    Http 01:
    Private Key Secret Ref:
      Key:
      Name:  letsencrypt
    Server:  https://acme-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:  https://acme-v02.api.letsencrypt.org/acme/acct/48899673
  Conditions:
    Last Transition Time:  2019-01-04T09:28:33Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

Describing the tls-secret certificate:

Name:         tls-secret
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"tls-secret","namespace":"default"},"spec":{"acme"...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Cluster Name:
  Creation Timestamp:  2019-01-04T09:28:13Z
  Resource Version:    130060
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/tls-secret
  UID:                 0f38w7y4-1003-11e9-997f-e6e9b4bcc625
Spec:
  Acme:
    Config:
      Domains:
        mydomain.com
      Http 01:
        Ingress Class:  nginx
  Dns Names:
    mydomain.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt
  Secret Name:  tls-secret
Events:         <none>

Describing the aks-ingress ingress controller:

Name:             aks-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
TLS:
  tls-secret terminates mydomain.com
Rules:
  Host                                                       Path  Backends
  ----                                                       ----  --------
  mydomain.com
                                                             /   myapplication:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:   ...
  kubernetes.io/ingress.class:                 nginx
  nginx.ingress.kubernetes.io/rewrite-target:  /
  certmanager.k8s.io/cluster-issuer:           letsencrypt
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  21m   nginx-ingress-controller  Ingress default/aks-ingress
  Normal  CREATE  21m   nginx-ingress-controller  Ingress default/aks-ingress

Logs for cert-manager after restarting the server:

I0104 09:28:38.378953 1 setup.go:144] Skipping re-verifying ACME account as cached registration details look sufficient.    
I0104 09:28:38.379058 1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt"   
I0104 09:28:38.378953 1 setup.go:144] Skipping re-verifying ACME account as cached registration details look sufficient.    
I0104 09:28:38.379058 1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt"   
I0104 09:28:38.378455 1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt'    
I0104 09:28:38.378455 1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt'    
I0104 09:28:33.440466 1 controller.go:185] certificates controller: Finished processing work item "default/tls-secret"  
I0104 09:28:33.440417 1 sync.go:206] Certificate default/tls-secret scheduled for renewal in 1423 hours 
I0104 09:28:33.440466 1 controller.go:185] certificates controller: Finished processing work item "default/tls-secret"  
I0104 09:28:33.440417 1 sync.go:206] Certificate default/tls-secret scheduled for renewal in 1423 hours 
I0104 09:28:33.439824 1 controller.go:171] certificates controller: syncing item 'default/tls-secret'   
I0104 09:28:33.439824 1 controller.go:171] certificates controller: syncing item 'default/tls-secret'   
I0104 09:28:33.377556 1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt"   
I0104 09:28:33.377556 1 controller.go:154] clusterissuers controller: Finished processing work item "letsencrypt"   
I0104 09:28:33.359246 1 helpers.go:147] Setting lastTransitionTime for ClusterIssuer "letsencrypt" condition "Ready" to 2019-01-04 09:28:33.359214315 +0000 UTC m=+79.014291591 
I0104 09:28:33.359178 1 setup.go:181] letsencrypt: verified existing registration with ACME server  
I0104 09:28:33.359178 1 setup.go:181] letsencrypt: verified existing registration with ACME server  
I0104 09:28:33.359246 1 helpers.go:147] Setting lastTransitionTime for ClusterIssuer "letsencrypt" condition "Ready" to 2019-01-04 09:28:33.359214315 +0000 UTC m=+79.014291591 
I0104 09:28:32.427832 1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt'    
I0104 09:28:32.427978 1 controller.go:182] ingress-shim controller: Finished processing work item "default/aks-ingress" 
I0104 09:28:32.427832 1 controller.go:140] clusterissuers controller: syncing item 'letsencrypt'    
I0104 09:28:32.427832 1 controller.go:168] ingress-shim controller: syncing item 'default/aks-ingress'  
I0104 09:28:32.428133 1 logger.go:88] Calling GetAccount    
I0104 09:28:32.427936 1 sync.go:140] Certificate "tls-secret" for ingress "aks-ingress" already exists  
I0104 09:28:32.427965 1 sync.go:143] Certificate "tls-secret" for ingress "aks-ingress" is up to date   
I0104 09:28:32.427978 1 controller.go:182] ingress-shim controller: Finished processing work item "default/aks-ingress" 
I0104 09:28:32.428133 1 logger.go:88] Calling GetAccount    
I0104 09:28:32.427936 1 sync.go:140] Certificate "tls-secret" for ingress "aks-ingress" already exists  
I0104 09:28:32.427832 1 controller.go:168] ingress-shim controller: syncing item 'default/aks-ingress'  
I0104 09:28:32.427965 1 sync.go:143] Certificate "tls-secret" for ingress "aks-ingress" is up to date   
I0104 09:28:29.439299 1 controller.go:171] certificates controller: syncing item 'default/tls-secret'   
E0104 09:28:29.439586 1 controller.go:180] certificates controller: Re-queuing item "default/tls-secret" due to error processing: Issuer letsencrypt not ready  
I0104 09:28:29.439404 1 sync.go:120] Issuer letsencrypt not ready   
E0104 09:28:29.439586 1 controller.go:180] certificates controller: Re-queuing item "default/tls-secret" due to error processing: Issuer letsencrypt not ready  
I0104 09:28:29.439299 1 controller.go:171] certificates controller: syncing item 'default/tls-secret'   
I0104 09:28:29.439404 1 sync.go:120] Issuer letsencrypt not ready   
I0104 09:28:27.404656 1 controller.go:68] Starting certificates controller  
I0104 09:28:27.404606 1 controller.go:68] Starting issuers controller   
I0104 09:28:27.404325 1 controller.go:68] Starting ingress-shim controller  
I0104 09:28:27.404606 1 controller.go:68] Starting issuers controller   
I0104 09:28:27.404325 1 controller.go:68] Starting ingress-shim controller  
I0104 09:28:27.404269 1 controller.go:68] Starting clusterissuers controller    
I0104 09:28:27.404656 1 controller.go:68] Starting certificates controller  
I0104 09:28:27.404269 1 controller.go:68] Starting clusterissuers controller    
I0104 09:28:27.402806 1 leaderelection.go:184] successfully acquired lease kube-system/cert-manager-controller  
I0104 09:28:27.402806 1 leaderelection.go:184] successfully acquired lease kube-system/cert-manager-controller  
I0104 09:27:14.359634 1 server.go:84] Listening on http://0.0.0.0:9402  
I0104 09:27:14.357610 1 controller.go:126] Using the following nameservers for DNS01 checks: [10.0.0.10:53] 
I0104 09:27:14.357610 1 controller.go:126] Using the following nameservers for DNS01 checks: [10.0.0.10:53] 
I0104 09:27:14.358408 1 leaderelection.go:175] attempting to acquire leader lease kube-system/cert-manager-controller...    
I0104 09:27:14.359634 1 server.go:84] Listening on http://0.0.0.0:9402  
I0104 09:27:14.356692 1 start.go:79] starting cert-manager v0.5.2 (revision 9e8c3ad899c5aafaa360ca947eac7f5ba6301035)   
I0104 09:27:14.358408 1 leaderelection.go:175] attempting to acquire leader lease kube-system/cert-manager-controller...    
I0104 09:27:14.356692 1 start.go:79] starting cert-manager v0.5.2 (revision 9e8c3ad899c5aafaa360ca947eac7f5ba6301035)

Certificate resource:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: tls-secret
spec:
  secretName: tls-secret
  dnsNames:
  - mydomain.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - mydomain.com
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer

i dont see cert-manager certificate resource? did you configure that? — 4c74356b41
@Rico I assumed that would have been deleted upon deleting cert-manager. Let me try. — Dave New
@4c74356b41 thanks for your help. I think I had to delete the secret associated with the certificate as well before recreating the certificate. All working now. — Dave New

4c74356b41 4c74356b41 · Accepted Answer · 2019-01-08T07:22:53

in this case the problem went away after recreating the secret and the cert-manager certificate resource.

generally what you want to check, annotations on your ingress resource (certmanager.k8s.io/cluster-issuer: letsencrypt), cert-manager certificate resource, ssl certificate secret in k8s and in ingress resource

Kubernetes cert-manager not updating certificates after issuer change

1 Answers