K3s ingress TLS enabled accessing TLS enabled backend, how to?

Question

I have a local K3s kubernetes cluster with its traefik ingress controller.

(Mac OSX, Multipass Hyper-V based local VMs: v1.18.3+k3s1 Ubuntu 16.04.6 LTS 4.4.0-179-generic containerd://1.3.3-k3s2)

what I want is having an ingress that is tls enabled AND forwarding to vault port 8200 via tls/https

k get -n kube-system svc traefik
NAME      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
traefik   LoadBalancer   10.43.105.6   192.168.64.5   80:30303/TCP,443:30142/TCP   4h21m

$ kubectl get all
NAME          READY   STATUS    RESTARTS   AGE
pod/vault-0   1/1     Running   0          4h31m

NAME                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
service/vault-internal   ClusterIP   None          <none>        8200/TCP,8201/TCP   4h31m
service/vault            ClusterIP   10.43.8.235   <none>        8200/TCP,8201/TCP   4h31m

NAME                     READY   AGE
statefulset.apps/vault   1/1     4h31m

I deployed vault via helm chart standalone vault (non dev mode) and tls enabled (values.yaml see below)

vault's cert is signed by k3s itself: kubectl -n "${NAMESPACE}" certificate approve "${CSR_NAME}"

certinfo tmp/localK3s/certs/vault/vault.crt
        Version: 3 (0x2)
        Serial Number:
            ed:8f:07:da:0d:3d:8d:55:3d:73:aa:93:9d:98:d2:69
    Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN=k3s-server-ca@1591718124
        Validity
            Not Before: Jun  9 15:53:56 2020 GMT
            Not After : Jun  9 15:53:56 2021 GMT
        Subject: CN=vault.vault.svc
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:vault, DNS:vault.vault, DNS:vault.vault.svc, DNS:vault.vault.svc.iac.local, DNS:localhost, IP Address:127.0.0.1

now I can access the vault service directly by e.g.:

$ kubectl -n vault port-forward service/vault 8200:8200 &
$
$ export VAULT_ADDR=https://127.0.0.1:8200
$ export VAULT_CAPATH=$(pwd)/tmp/localK3s/certs/localK3s_root.ca
$ export VAULT_CLIENT_CERT=$(pwd)/tmp/localK3s/certs/vault/vault.crt
$ export VAULT_CLIENT_KEY=$(pwd)/tmp/localK3s/certs/vault/vault.key
$ 
$ vault status
Handling connection for 8200
Handling connection for 8200
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.4.2
Cluster Name    vault-cluster-5bc9e954
Cluster ID      ca5496a6-525d-2b86-22dd-f771da82d5e0
HA Enabled      false

now what I want is having an ingress that is tls enabled AND forwarding to vault port 8200 via tls/https

so I have

$ kubectl get ingress vault -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: vault
  namespace: vault
  annotations:
    meta.helm.sh/release-name: vault
    meta.helm.sh/release-namespace: vault
  labels:
    helm.sh/chart: vault-0.6.0
spec:
  rules:
  - host: vault.iac.local
    http:
      paths:
      - backend:
          serviceName: vault
          servicePort: 8200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - vault.iac.local
    secretName: vault-tls
status:
  loadBalancer: {}

$ export VAULT_ADDR=https://vault.iac.local
$ export VAULT_CAPATH=$(pwd)/tmp/localK3s/certs/localK3s_root.ca
$ export VAULT_CLIENT_CERT=$(pwd)/tmp/localK3s/certs/vault/vault.crt
$ export VAULT_CLIENT_KEY=$(pwd)/tmp/localK3s/certs/vault/vault.key
$
$ vault status
vault status -tls-skip-verify
Error checking seal status: Error making API request.

URL: GET https://vault.iac.local/v1/sys/seal-status
Code: 404. Raw Message:

404 page not found

helm vault values.yaml

global:
  enabled: true
  tlsDisable: false
injector:
  enabled: false
server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
  extraVolumes:
  - type: secret
    name: vault-tls
  standalone:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = "false" # 1
        # address = "0.0.0.0:8200"
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file =      "/vault/userconfig/vault-tls/vault.crt"
        tls_key_file  =      "/vault/userconfig/vault-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
      }
      storage "file" {
        path = "/vault/data"
      }
  ingress:
    enabled: true
    hosts:
    - host: vault.iac.local
    tls:
    - secretName: vault-tls
      hosts:
      - vault.iac.local

any ideas anybody?

Does the same happen when you deploy cert-manager, configure issuer and create a cert? learn.hashicorp.com/vault/kubernetes/… — Mariusz K.

Dirk Hoffmann Dirk Hoffmann · Accepted Answer · 2020-06-11T09:46:24

ok, it's always helpful to read the logs (sigh), e.g. of the ingress controller itself, e.g.:

INGCTRL=traefik && \
kubectl -n kube-system logs \
    pod/$(kubectl -n kube-system get pods -l app=$INGCTRL | sed -n -E "s/^($INGCTRL-[a-z0-9-]+).*$/\1/p")

if you use generic secrets for ingress tls, beware the secret keys have to be tls.crt and tls.key (or use kubectl create secret tls and not generic in the first place)

also check that your target service has an endpoint at all and not

k describe svc theService
...
Endpoints:         <none>
...

K3s ingress TLS enabled accessing TLS enabled backend, how to?

1 Answers