How to debug Kubernetes nginx Ingress redirection from HTTP to HTTPS

Question

Fast question: how I can debug ingress and Nginx to know where exactly HTTP->HTTPS redirection happens?

More details:

What we have: we have war file + Tomcat, build it with Docker. Run it with Kubernetes in AWS.

What we need: the application should be accessible with HTTP and with HTTPS. HTTP should not redirect to HTTPS.

Problem: HTTP always redirects to HTTPS.

What we try: we have Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ${some name
  namespace: ${some namespace}
  labels:
    app: ${some app}
    env: ${some env}
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false" #we added this for turn off https redirection
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false" #we added this for turn off https redirection
    nginx.ingress.kubernetes.io/affinity: "cookie" # We use it for sticky sessions
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "some cookie name"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/whitelist-source-range: ${whitelist of ip adresses}

spec:
  tls:
    - hosts:
        - ${some host}
        - ${some another host}
      secretName: my-ingress-ssl
  rules:
    - host: ${some host}
      http:
        paths:
          - path: /
            backend:
              serviceName: ${some another service name}
              servicePort: 8080
    - host: ${some another host}
      http:
        paths:
          - path: /
            backend:
              serviceName: ${some another service name}
              servicePort: 8080

And configmap

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    app: ${some app}
    env: ${some env}
  namespace: ${some namespace}
  name: nginx-config
data:
  hsts: "false" #we added this for turn off https redirection
  hsts-max-age: "0" #we added this for turn off https redirection
  ssl-redirect: "false" #we added this for turn off https redirection
  hsts-include-subdomains: "false" #we added this for turn off https redirection

In Tomcat server.xml we have:

<Connector port="8080" protocol="HTTP/1.1"
                   connectionTimeout="20000"
                   redirectPort="8443" />

...

<!-- Define an AJP 1.3 Connector on port 8009 -->
        <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

...

and this connector we commented (it shouldn't work now):

<!--
<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
           maxThreads="150" SSLEnabled="true" >
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
    <SSLHostConfig>
        <Certificate certificateKeyFile="conf/key.pem"
                     certificateFile="conf/cert.pem"
                     certificateChainFile="conf/chain.pem"
                     type="RSA" />
    </SSLHostConfig>
</Connector>
-->

I tried all possible variants with ingress annotations, but without success result.

What I want to know: how can I debug ingress with Nginx to know where exactly HTTP->HTTPS redirection happens?

UPD

As it turned out, it was not the Nginx controller that was installed on the server, but Traefik. Due to security restrictions, I cant see that pod with the controller. So no Nginx annotations worked. Nevertheless, the answers below to my questions will help people with a similar problem.

Have you ever had HSTS configured for the domain that you are testing? Make sure to clear all HSTS config for your browser before testing, and use a fresh anonymous tab...If you have an HSTS entry set on your browser, the SSL will be forced on the client-side by an internal redirect (HTTP status code 307). — Eduardo Baitello
We have tested it in anonymous new tab and received 301 status — Anton Rey
@EduardoBaitello we also call that URL from Java without any cookies and settings. Received 301 — Optio
Are you using the kubernetes/ingress-nginx or the nginxinc/kubernetes-ingress image for your controller? Which version? Can you check if the --annotations-prefix matches the annotations that you are using? Can you share how your ingress service is configured? Do you have Load Balancers in front of it? (ELB (Classic)? ALB? NLB?) — Eduardo Baitello
You can do curl -vvv to the http port and check what is the server in the redirect reply header, it should give you an idea who is responsible for redirecting. — Ottovsky

Richard Woods Richard Woods · Accepted Answer · 2020-07-21T01:27:53

In the setup you describe there are likley to be at least three processes involved in the handling of incoming http requests

internet -> AWS ELB -> Nginx Ingress Controller -> Tomcat

Any of the processes may be able to issue a 301 redirect to instruct http traffic to retry with https. A redirect in the tomcat process could either be specified by tomcat configuration or the app tomcat hosts.

I would attempt to isolate the process that is performing the redirect by experimenting to bypass the higher processes:

Bypass the loadbalancer. SSH onto one of the K8s nodes and curl the ingress services' nodeport in the URL (check out Internal Endpoints on the nginx-controller service description for the right port). If you get a redirect, you know the problem is not the the loadbalancer.
Bypass the ingress controllers by 'docker exec' into one of the app containers, then curl localhost:8080. If you get a redirect, you know the problem is not the the ingress controller, and must be in the tomcat config or application.

If none of this is possible due to ssh restrictions, curl availability etc... A different approach would be to turn on request logging for each of the processes involved - though this will not be appropriate for a production workload as it will lead to confidential information being output to inappropriate contexts, such as a web tier log file or cloudwatch logs.

How to debug Kubernetes nginx Ingress redirection from HTTP to HTTPS

2 Answers