Let's Encrypt Certificate Issuance

3
votes

I am trying to get a certificate issued from Let's Encrypt, and it has been 3 and a half hours.

I accidentally originally set my secretName as "echo-tls" before switching it to the correct "pandaist-tls" that I want to use instead.

I currently have this:

kubectl get CertificateRequest -o wide
NAME                      READY   ISSUER             STATUS                                                                                             AGE
pandaist-tls-1926992011   False   letsencrypt-prod   Waiting on certificate issuance from order default/pandaist-tls-1926992011-2163900139: "pending"   3h26m

When I describe the certificate, I get this:

Deployment kubectl describe CertificateRequest pandaist-tls-1926992011
Name:         pandaist-tls-1926992011
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: pandaist-tls
              cert-manager.io/private-key-secret-name: pandaist-tls
API Version:  cert-manager.io/v1alpha2
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2020-04-07T15:41:13Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  pandaist-tls
    UID:                   25c3ff31-447f-4abf-a23e-ec48f5a591a9
  Resource Version:        500795
  Self Link:               /apis/cert-manager.io/v1alpha2/namespaces/default/certificaterequests/pandaist-tls-1926992011
  UID:                     8295836d-fb99-4ebf-8803-a344d6edb574
Spec:
  Csr:  ABUNCHOFVALUESTHATIWILLNOTDESCRIBE
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-prod
Status:
  Conditions:
    Last Transition Time:  2020-04-07T15:41:13Z
    Message:               Waiting on certificate issuance from order default/pandaist-tls-1926992011-2163900139: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:                    <none>

And then I look at my logs for my cert-manager pods - here are small slices of each:

I0407 19:01:35.499469       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-2fp58" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01" 
I0407 19:01:35.499513       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="auth.pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-xhjsr" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01" 
I0407 19:01:35.499534       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-pd9fh" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01" 
I0407 19:01:35.499578       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="auth.pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-b6zr2" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01" 
E0407 19:03:46.571074       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://pandaist.com/.well-known/acme-challenge/6Wduj2Ejr59OZ9SFy_Rw4jnozE50xspK-a5OIvCwYsc': Get http://pandaist.com/.well-known/acme-challenge/6Wduj2Ejr59OZ9SFy_Rw4jnozE50xspK-a5OIvCwYsc: dial tcp 178.128.132.218:80: connect: connection timed out" "dnsName"="pandaist.com" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01" 
E0407 19:03:46.571109       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://auth.pandaist.com/.well-known/acme-challenge/gO91--fK0SGG15aS3ALOHXXYtCSly2Q9pbVO8OJW2aE': Get http://auth.pandaist.com/.well-known/acme-challenge/gO91--fK0SGG15aS3ALOHXXYtCSly2Q9pbVO8OJW2aE: dial tcp 178.128.132.218:80: connect: connection timed out" "dnsName"="auth.pandaist.com" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01" 
I0407 19:03:46.571382       1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/pandaist-tls-1926992011-2163900139-832917849" 
I0407 19:03:46.571528       1 controller.go:129] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="default/pandaist-tls-1926992011-2163900139-832917849" 
I0407 19:03:46.571193       1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/pandaist-tls-1926992011-2163900139-2157075729" 
I0407 19:03:46.572009       1 controller.go:129] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="default/pandaist-tls-1926992011-2163900139-2157075729" 
I0407 19:03:46.572338       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="auth.pandaist.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-scqtx" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01" 
I0407 19:03:46.572600       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="auth.pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-xhjsr" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01" 
I0407 19:03:46.572860       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="auth.pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-b6zr2" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01" 
I0407 19:03:46.573128       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="pandaist.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-jn65v" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01" 
I0407 19:03:46.573433       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-2fp58" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01" 
I0407 19:03:46.573749       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-pd9fh" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"

And then here, where I still see echo-tls, despite the fact that I changed my ingress to use pandaist-tls:

I0407 15:34:37.115159       1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0407 15:34:37.118246       1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0407 15:34:37.118520       1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0407 15:34:37.119415       1 sources.go:176] cert-manager/inject-controller "level"=0 "msg"="Extracting CA from Secret resource" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-tls"
I0407 15:34:37.120959       1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0407 15:34:37.121399       1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0407 15:34:37.124545       1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0407 15:34:37.125160       1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
E0407 16:19:36.762436       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"} 
E0407 16:19:36.762573       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"} 
E0407 16:19:36.762753       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"} 
E0407 16:19:36.762766       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"}

My ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: pandaist-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - pandaist.com
    - auth.pandaist.com
    secretName: pandaist-tls
  rules:
  - host: pandaist.com
    http:
      paths:
      - backend:
          serviceName: pandaist-main
          servicePort: 80
  - host: auth.pandaist.com
    http:
      paths:
      - backend:
          serviceName: pandaist-keycloak
          servicePort: 80

This ingress was absolutely applied after the echo one.

Is this just normal certificate approval time (3.5 hours) or did the accidental inclusion of echo-tls mess up my certificate issuance? If so, how do I fix it?

2
sslkubernetesssl-certificate

2 Answers

2
votes

Due to a bug in how load balancers work on Digital Ocean:

https://www.digitalocean.com/community/questions/how-do-i-correct-a-connection-timed-out-error-during-http-01-challenge-propagation-with-cert-manager

This will solve the problem:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https
1
votes

This might be worthwhile to look at in case @Cecil's solution doesn't work. I was facing similar issue with Connection Timeout

Change LoadBalancer in ingress-nginx service.

Add/Change externalTrafficPolicy: Cluster.

Reason being, pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress.

Below is complete block taken from https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  #CHANGE/ADD THIS
  externalTrafficPolicy: Cluster
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

---