error broken header: "GET /.well-known/acme-challeng with LetsEncrypt on Kubernates

1
votes

I've created the LetsEncrypt production ClusterIssuers in Digital Ocean Kubernaties DO kubernaties ver - 1.17.5 My cert-manager version is v0.15.0

I used this howto

kubectl describe clusterissuer letsencrypt-prod

Name:         letsencrypt-prod
Namespace:    
Labels:       <none>
Annotations:  API Version:  cert-manager.io/v1alpha3
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2020-05-13T12:08:52Z
  Generation:          1
  Resource Version:    16757
  Self Link:           /apis/cert-manager.io/v1alpha3/clusterissuers/letsencrypt-prod
  UID:                 2bbd1ca6-9c85-45e3-ad6e-7b85d9e93657
Spec:
  Acme:
    Email:  [email protected]
    Private Key Secret Ref:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  [email protected]
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/86033097
  Conditions:
    Last Transition Time:  2020-05-13T12:08:53Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

kubectl describe ingress

Name:             bb-ingress
Namespace:        default
Address:          167.99.17.96
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  bb-cloud-tls terminates example.com
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  example.com  
                       /   bb-web-service:80 (10.244.0.166:3000,10.244.0.31:3000)
Annotations:           cert-manager.io/cluster-issuer: letsencrypt-prod
                       kubernetes.io/ingress.class: nginx
Events:
  Type     Reason     Age                   From                      Message
  ----     ------     ----                  ----                      -------
  Warning  BadConfig  8m17s                 cert-manager              TLS entry 0 for hosts [example.com] must specify a secretName
  Normal   UPDATE     7m24s (x11 over 24h)  nginx-ingress-controller  Ingress default/bb-ingress


Name:             cm-acme-http-solver-kbnn6
Namespace:        default
Address:          167.99.17.96
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  example.com  
                       /.well-known/acme-challenge/i5J8QI4XwJZVnS4xC_nSbK-8QFYlUJkmmOnETFXltdE   cm-acme-http-solver-kgbd8:8089 (10.244.0.188:8089)
Annotations:           kubernetes.io/ingress.class: nginx
                       nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events:                <none>

kubectl describe certificate

Name:         bb-cloud-tls
Namespace:    default
Labels:       <none>
Annotations:  API Version:  cert-manager.io/v1alpha3
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-05-13T11:06:34Z
  Generation:          1
  Resource Version:    13723
  Self Link:           /apis/cert-manager.io/v1alpha3/namespaces/default/certificates/bb-cloud-tls
  UID:                 11e6d711-56a9-4711-a6c4-cca516b96c41
Spec:
  Common Name:  example.com
  Dns Names:
    example.com
  Duration:  24h0m0s
  Issuer Ref:
    Kind:        ClusterIssuer
    Name:        letsencrypt-prod
  Renew Before:  12h0m0s
  Secret Name:   bb-cloud-tls
Status:
  Conditions:
    Last Transition Time:  2020-05-13T11:46:24Z
    Message:               Waiting for CertificateRequest "bb-cloud-tls-1534494017" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:                    <none>

kubectl describe order

Name:         bb-cloud-tls-1534494017-2165728012
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: bb-cloud-tls
              cert-manager.io/private-key-secret-name: bb-cloud-tls
API Version:  acme.cert-manager.io/v1alpha3
Kind:         Order
Metadata:
  Creation Timestamp:  2020-05-13T11:46:24Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  bb-cloud-tls-1534494017
    UID:                   5b2972ba-bfe5-4149-a53b-13764a1a8269
  Resource Version:        13730
  Self Link:               /apis/acme.cert-manager.io/v1alpha3/namespaces/default/orders/bb-cloud-tls-1534494017-2165728012
  UID:                     1dd81160-c700-4d29-88c1-0c5a5dee5774
Spec:
  Common Name:  example.com
  Csr:          LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNU**************************
  Dns Names:
    example.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prod
Status:
  Authorizations:
    Challenges:
      Token:        i5J8QI4XwJZVnS4*********
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4557349440/4vbwhw
      Token:        i5J8QI4XwJZVnS******
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4557349440/yILvmw
      Token:        i5J8QI4Xw*****
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4557349440/iPGc-Q
    Identifier:     example.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/4557349440
    Wildcard:       false
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/86033097/3348998322
  State:            pending
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/86033097/3348998322
Events:             <none>

Also I have such logs for ingress pod devspace logs -n ingress-nginx --pod ingress-nginx-controller-5cc4589cc8-z5hb4 -c controller

" while reading PROXY protocol, client: 10.244.0.178, server: 0.0.0.0:80
2020/05/14 11:59:02 [error] 163#163: *388536 broken header: "GET /.well-known/acme-challenge/i5J8QI4XwJZVnS4xC_nSbK-8QFYlUJkmmOnETFXltdE HTTP/1.1
Host: example.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
Connection: close

I have incorrect certificate: "Kubernetes Ingress Controller Fake Certificate"

How I can fix this issue?

PS. also I found simular issue on githib but it is closed and I have new version of cert-manager

2
digital-oceankubernetes-ingress
Don't use http01, but rather the dns self-check or use one of the workarounds. One was hardcoding some ips I'm not certain what option it exactly was. – Fabian
How it is possible? How to replace http01? I am newbie in kubernates – mpz
Just replace your solver inside your cluster issuer with a dns-solver. Thats at least how it worked for me. Its name above is letsencrypt-prod. – Fabian
Seems you said about ACME of cert-manager: cert-manager.io/docs/configuration/acme/#solving-challenges so I need to replace my "Solving Challenges" from HTTP01 to DNS01 – mpz
Yes. Scroll down to > DNS names – Fabian

2 Answers

2
votes

I change ACME from http01 to dns01

before:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

after:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the DNS-01 challenge provider
    solvers:
    - dns01:
        digitalocean:
          tokenSecretRef:
            name: digitalocean-dns
            key: access-token

Also I add Secret - see https://cert-manager.io/docs/configuration/acme/dns01/digitalocean/ for details

Now it is works

0
votes

@mpz Please refer to this issue : https://github.com/jetstack/cert-manager/issues/466

Note one of the comments states "Unfortunately the DNS01 challenge is broken for DigitalOcean in 0.7.0 (and based on my testing in 0.6.0 as well) so HTTP01 is a must for DO." , Which is the opposite of your answer . I'm not sure if this is fixed or not, but I was able to fix this specific issue and get HTTP01 Challenge working with compumike's answer https://github.com/compumike/hairpin-proxy . It explains the issue around problem, and presents a simple fix as a one-line install (that should work out of the box with ingress-nginx and cert-manager) .

Another recent answer by KeksBeskvitovich (That I didn't attempt) was a DigitalOcean specific annotation to the Ingress Controller Service 'service.beta.kubernetes.io/do-loadbalancer-hostname' ( https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname ) . Assuming this works (again I haven't tried this yet), this would be a more official solution as it doesn't require the 3rd party installation.

But Compumike's hairpin proxy solution was simple, easy, and worked for me (was the final piece of the puzzle), so if you're struggling with certmanager, give this one a try!