Applying TLS to a working ingress results in default backend - 404

Question

I have installed the Nginx Ingress Controller through helm in the ingress namespace.

helm ls --namespace ingress
NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                   APP VERSION
nginx-ingress   ingress     1           2020-03-15 10:47:51.143159 +0530 IST    deployed    nginx-ingress-1.34.2    0.30.0  

The Service and Deployment is as follows

apiVersion: v1
kind: Service
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: test-service
    app.kubernetes.io/instance: test-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: test-service
      app.kubernetes.io/instance: test-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: test-service
        app.kubernetes.io/instance: test-service
    spec:
      containers:
        - name: test-service
          image: "<acr-url>/test-service:c93c58c0bd4918de06d46381a89b293087262cf9"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /devops/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /devops/health/readiness
              port: 8080
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          env:
            - name: test-serviceClientId
              valueFrom:
                secretKeyRef:
                  key: test-serviceClientId
                  name: test-service-133
            - name: test-serviceClientSecret
              valueFrom:
                secretKeyRef:
                  key: test-serviceClientSecret
                  name: test-service-133
            - name: test-serviceTenantClientId
              valueFrom:
                secretKeyRef:
                  key: test-serviceTenantClientId
                  name: test-service-133
            - name: test-serviceTenantClientSecret
              valueFrom:
                secretKeyRef:
                  key: test-serviceTenantClientSecret
                  name: test-service-133
          resources:
            limits:
              cpu: 800m
            requests:
              cpu: 300m

Configured the ingress on a service with rewrite as follows

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
    - apiexample.centralus.cloudapp.azure.com
    secretName: tls-secret
  rules:
    - host: "apiexample.centralus.cloudapp.azure.com"
      http:
        paths:
          - path: /testservice(/|$)(.*)
            backend:
              serviceName: test-service
              servicePort: 8080

The tls-secret has been generated using

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"

$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt

Before applying the tls configuration in the ingress, I was able to get response from the api endpoint. The api endpoint is secured with oauth.

API Endpoint:

http://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint

After applying the TLS config on the ingress, and hitting

https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint

I am getting default backend 404.

I have tested the TLS with ingress using another sample service (which is not secured with oauth) and it seems to be working for that service. Here's the configuration for the other services

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tea
  template:
    metadata:
      labels:
        app: tea
    spec:
      containers:
      - name: tea
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: tea

The ingress for the service is configured as follows

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
    - apiexample.centralus.cloudapp.azure.com
    secretName: tls-secret
  rules:
  - host: apiexample.centralus.cloudapp.azure.com
    http:
      paths:
      - path: /teaprefix(/|$)(.*)
        backend:
          serviceName: tea-svc
          servicePort: 80

The endpoint

https://apiexample.centralus.cloudapp.azure.com/teaprefix/someurl

works fine.

Please let me know if there's anything missing in my configuration and any potential issues that i may have ignored.

Note: The Service and the Ingress are deployed in the default namespace and the Ingress Controller is running in the ingress namespace The Nginx Ingress Controller is running in 2 Pods

Logs from Ingress Controller with TLS configuration

Pod1

 10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 302 0 "-" "PostmanRuntime/7.23.0" 1495 0.004 [default-test-service-8080] [] 10.244.0.7:8080 0 0.004 302 f4671ede2f95148220c21fe44de6fdad
 10.244.0.1 - - [22/Mar/2020:06:57:13 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 404 21 "http://apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1563 0.001 [upstream-default-backend] [] 10.244.0.225:8080 21 0.004 404 ed41b36bc6b89b60bc3f208539a0d44c

Pod2

    10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 308 171 "https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1580 0.000 [upstream-default-backend] [] - - - - ce955b7bb5118169e99dd4051060c897

Logs from Ingress Controller without TLS configuration

10.244.0.1 - - [22/Mar/2020:07:04:34 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 200 276 "-" "PostmanRuntime/7.23.0" 1495 2.165 [default-test-service-8080] [] 10.244.0.4:8080 548 2.168 200 e866f277def90c398df4e509e45718b2

UPDATE

Disabling the authentication on the backend service (test-service) also results in the same behavior. Without applying TLS, able to hit the endpoint using http without any Bearer Token. After applying TLS, get a default backend - 404 when i hit the endpoint with https/http

UPDATE

Exposing the Service via ClusterIP without the

service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'

annotation instead of LoadBalancer also does not seem to be helping. The endpoint works without TLS and with TLS applied, get a default backend - 404

UPDATE

The test-service is a Spring Boot Application with the following WebSecurityConfiguration

@Component
@EnableResourceServer
public class WebSecurityConfiguration extends ResourceServerConfigurerAdapter {

  private static final Logger LOGGER = LoggerFactory.getLogger(WebSecurityConfiguration.class);

  private final HealthCheckWebSecurity healthCheckWebSecurity = new HealthCheckWebSecurity();

  private final Oauth2Settings oauth2Settings;
  private final JwtTokenStore jwtTokenStore;
  private final TenantService tenantService;
  private final TransportGuaranteeWebSecurity transportGuaranteeWebSecurity;

  @Autowired
  public WebSecurityConfiguration(
      Oauth2Settings oauth2Settings,
      JwtTokenStore jwtTokenStore,
      TenantService tenantService,
      TransportGuaranteeWebSecurity transportGuaranteeWebSecurity) {
    this.oauth2Settings = oauth2Settings;
    this.jwtTokenStore = jwtTokenStore;
    this.tenantService = tenantService;
    this.transportGuaranteeWebSecurity = transportGuaranteeWebSecurity;
  }

  @Override
  public void configure(ResourceServerSecurityConfigurer resources) throws Exception {
    String resourceId = oauth2Settings.getResource("default").getResourceId();
    LOGGER.info("Resource service id: {}", resourceId);

    resources.resourceId(resourceId).tokenStore(jwtTokenStore);
  }

  @Override
  public void configure(HttpSecurity http) throws Exception {
    http.requestMatchers().anyRequest();
    http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
    http.csrf().disable();

    healthCheckWebSecurity.configure(http);
    transportGuaranteeWebSecurity.configure(http);

    http.authorizeRequests().anyRequest().permitAll();
    http.addFilterAfter(buildTenancyContextFilter(), ChannelProcessingFilter.class);
    http.addFilterAfter(buildLongUrlFilter(), ChannelProcessingFilter.class);
  }

  private TenancyContextFilter buildTenancyContextFilter() {
    return new TenancyContextFilter(tenantService,
        new PathVariableTenantExtractor(Arrays.asList("/{tenantAlias}/api/**")));
  }

  private LongRequestHttpFilter buildLongUrlFilter() {
    return new LongRequestHttpFilter();
  }
}

public final class TransportGuaranteeWebSecurity {

  private TransportGuaranteeSettings transportGuaranteeSettings;

  TransportGuaranteeWebSecurity(TransportGuaranteeSettings transportGuaranteeSettings) {
    this.transportGuaranteeSettings = transportGuaranteeSettings;
  }


  public void configure(HttpSecurity httpSecurity) throws Exception {
    if (httpsRequired()) {
      httpSecurity.requiresChannel().anyRequest().requiresSecure();
    } else {
      httpSecurity.requiresChannel().anyRequest().requiresInsecure();
    }
  }

  private boolean httpsRequired() {
    final String transportGuarantee = transportGuaranteeSettings.getTransportGuarantee();
    return !TransportGuaranteeSettings.TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee);
  }

}


@ConfigurationProperties(prefix = "web.security")
public class TransportGuaranteeSettings {
  static final String TRANSPORT_GUARANTEE_NONE = "NONE";
  static final String TRANSPORT_GUARANTEE_CONFIDENTIAL = "CONFIDENTIAL";

  private static final Logger LOGGER = LoggerFactory.getLogger(TransportGuaranteeSettings.class);

  private static final String TRANSPORT_GUARANTEE_PROPERTY = "web.security.transportGuarantee";

  private String transportGuarantee;

  public String getTransportGuarantee() {
    return transportGuarantee;
  }

  public void setTransportGuarantee(String transportGuarantee) {
    this.transportGuarantee = transportGuarantee.trim();
    logUnexpectedValue();
  }

  private void logUnexpectedValue() {
    if (!TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee)
        && !TRANSPORT_GUARANTEE_CONFIDENTIAL.equalsIgnoreCase(transportGuarantee)) {
      LOGGER.debug(
          "Unknown value '{}' for property '{}' (expected '{}' or '{}'). Defaulted to '{}'.",
          transportGuarantee, TRANSPORT_GUARANTEE_PROPERTY, TRANSPORT_GUARANTEE_NONE, TRANSPORT_GUARANTEE_CONFIDENTIAL,
          TRANSPORT_GUARANTEE_CONFIDENTIAL);
    }
  }
}

In my application.yaml,

web.security.transportGuarantee: NONE

The tenancy context filter extracts the Tenant information from the URL and sets a ThreadLocal. There should not be any issue with that since I am able to hit the endpoint without the TLS configuration. I also do not see any issue with the TransportGuaranteeWebSecurity for the same reason.

Some more logs for Debug

kubectl get pods -owide --namespace ingress

NAME                                             READY   STATUS    RESTARTS   AGE   IP             NODE                                NOMINATED NODE   READINESS GATES
nginx-ingress-controller-5fcbccd545-bdh25        1/1     Running   1          15d   10.244.0.22    aks-agentpool-44086776-vmss000000   <none>           <none>
nginx-ingress-controller-5fcbccd545-ptx6j        1/1     Running   0          15d   10.244.0.21    aks-agentpool-44086776-vmss000000   <none>           <none>
nginx-ingress-default-backend-554d7bd77c-zxzlf   1/1     Running   0          15d   10.244.0.225   aks-agentpool-44086776-vmss000000   <none>           <none>

kubectl get svc

test-service                          LoadBalancer   10.0.231.35    13.89.111.39    8080:31534/TCP               14d
tea-svc                                   ClusterIP      10.0.12.216    <none>          80/TCP                       17d

kubectl get ing

test-service         apiexample.centralus.cloudapp.azure.com   10.240.0.4   80, 443      15d

Answer 1

I've reproduced your scenario in my GCP account, and didn't get the same result, so I'm posting my steps to make the troubleshoot of each components in order to make sure all of them is working properly. In resume, seem's the main problem is how the application is handle the paths or host.

Kubernetes: 1.15.3 (GKE) Nginx Ingress: Installed following the offical docs

Based on your yaml, I removed the readiness and liveness probes, and environment envs to test, and changed the image to a nginx image (on port 80):

apiVersion: v1
kind: Service
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: test-service
    app.kubernetes.io/instance: test-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: test-service
      app.kubernetes.io/instance: test-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: test-service
        app.kubernetes.io/instance: test-service
    spec:
      containers:
        - name: test-service
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 80
              protocol: TCP

After applied we can check if both (deployment and service) are running as expected, before apply ingress spec.

To check this, we can use a curl image to curl destination or dnsutil container by oficial kubernetes docs. In this case I've used curlimages/curl do test:

apiVersion: v1
kind: Pod
metadata:
  name: curl
  namespace: default
spec:
  containers:
  - name: curl
    image: curlimages/curl
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

With curl container running we can check first if the container of our nginx image is correctly running and answers the requests 'curling' directly their IP.

The command below will create a variable named $pod from the pod with label app.kubernetes.io/name=test-service with the ip of the pod.

$ pod=$(kubectl get pods -ojsonpath='{.items[*].status.podIP}' -l app.kubernetes.io/name=test-service)

$ echo $pod
192.168.109.12

Using curl pod create earlier, we can check if the pod is processing the requests:

$ kubectl exec curl -- curl -Is $pod

HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Tue, 24 Mar 2020 09:08:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
Connection: keep-alive
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

See the response HTTP/1.1 200 OK, let's move forward to test the service:

$ kubectl exec curl -- curl -Is test-service
HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Tue, 24 Mar 2020 09:11:13 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
Connection: keep-alive
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

Same here, HTTP/1.1 200 OK for service.

Let's go further and deploy the nginx ingress without TLS yet, to make a test before and after:

Generating and applying the certificate:

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"
...
$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created

Ingress without TLS (I've changed to port 80 to match with my nginx image):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: "apiexample.centralus.cloudapp.azure.com"
      http:
        paths:
          - path: /testservice(/|$)(.*)
            backend:
              serviceName: test-service
              servicePort: 80

Testing from my desktop to internet using the IP (omitted) provided by GCP:

$ curl -ILH "Host: apiexample.centralus.cloudapp.azure.com" http://34.77.xxx.xx/testservice
HTTP/1.1 200 OK
Server: nginx/1.17.8
Date: Tue, 24 Mar 2020 10:41:21 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

Until here everything is working fine. We can now add the TLS to ingress spec and try again:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
    - apiexample.centralus.cloudapp.azure.com
    secretName: tls-secret
  rules:
    - host: "apiexample.centralus.cloudapp.azure.com"
      http:
        paths:
          - path: /testservice(/|$)(.*)
            backend:
              serviceName: test-service
              servicePort: 80

Testing using curl:

curl -ILH "Host: apiexample.centralus.cloudapp.azure.com" https://34.77.147.74/testservice -k
HTTP/2 200 
server: nginx/1.17.8
date: Tue, 24 Mar 2020 10:45:25 GMT
content-type: text/html
content-length: 612
vary: Accept-Encoding
last-modified: Tue, 03 Mar 2020 14:32:47 GMT
etag: "5e5e6a8f-264"
accept-ranges: bytes
strict-transport-security: max-age=15724800; includeSubDomains

Ok, so that is working with TLS, so based on this we can conclude that your yaml spec is working, and maybe your are facing some issue with path in ingress definitions and your application:

You are using the annotation nginx.ingress.kubernetes.io/rewrite-target: /$2 and in the path /testservice(/|$)(.*)

It means that, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation.

Based on your ingress path regex:

• apiexample.centralus.cloudapp.azure.com/testservice rewrites to apiexample.centralus.cloudapp.azure.com/

• apiexample.centralus.cloudapp.azure.com/testservice/ rewrites to apiexample.centralus.cloudapp.azure.com/

• apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint rewrites to apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint

When checked in nginx pod logs, you could see the requested url after rewrite:

2020/03/24 10:59:33 [error] 7#7: *186 open() "/usr/share/nginx/html/tenant/api/v1/endpoint" failed (2: No such file or directory), client: 10.20.1.61, server: localhost, request: "HEAD /tenant/api/v1/endpoint HTTP/1.1", host: "apiexample.centralus.cloudapp.azure.com"

So, by this test I could concluded that your deployment, service and ingress is working and doesn't has any typo or formatting problems. So my advice is to double check the application

• Make sure your application is handle correctly the path;
• If your applications is doing some URL validation, make sure it can handle http and https;
• In case you have CORS enabled, adjust the ingress as mentioned here.

Since you didn't post any app as example to reproduce, my tests was limited a generic app as backend. If you could provide more details about backend application, or some generic app which produces the same behavior please let me know and will happy to improve my answer with more details.

References:

https://kubernetes.github.io/ingress-nginx/deploy/

https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/

https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/