Why does my bare-metal kubernetes nginx Ingress-controller return a 308?

11
votes

I am trying to create add the nginx ingress controller to my kubernetes cluster. My current cluster has 3 nodes and they all have open firewall rules in between them. (Note: This is a bare metal cluster for educational purposes)

I have created a deployment with the following config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodejs-test
  labels:
    app: nodejs
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nodejs
  template:
    metadata:
      labels:
        app: nodejs
    spec:
      containers:
      - name: nodejs-container
        image: gcr.io/infrastructure/test-nodejs-server
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: test-deployment-service
spec:
  selector:
    app: nodejs
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000

After running kubectl create -f file.yaml

I see all pods and services running. Within the cluster I can curl that service ip and get a response from the pods.

Now I go to create a ingress.

I first followed the install steps found at https://github.com/kubernetes/ingress-nginx/tree/master/deploy

I can see my ingress controller running. 

    NAMESPACE       NAME                                         READY     STATUS    RESTARTS   AGE
default         my-test-6445d9bb7b-blm7t                     1/1       Running   0          3h
default         nodejs-test-5657f7bb74-7hwqk                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-bh44x                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-dw4g4                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-k7bf4                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-xdthl                 1/1       Running   0          23h
ingress-nginx   default-http-backend-55c6c69b88-4d5bf        1/1       Running   0          26m
ingress-nginx   nginx-ingress-controller-5c6698dfbf-gs9zd    1/1       Running   0          25m
kube-system     etcd-yoda.me.com                      1/1       Running   1          1d
kube-system     kube-apiserver-yoda.me.com            1/1       Running   1          1d
kube-system     kube-controller-manager-yoda.me.com   1/1       Running   1          1d
kube-system     kube-dns-6f4fd4bdf-62bxg                     3/3       Running   3          1d
kube-system     kube-flannel-ds-jbmwg                        1/1       Running   1          1d
kube-system     kube-flannel-ds-r8nfb                        1/1       Running   2          1d
kube-system     kube-flannel-ds-rczbx                        1/1       Running   0          1d
kube-system     kube-proxy-brm2g                             1/1       Running   0          1d
kube-system     kube-proxy-qjgcv                             1/1       Running   1          1d
kube-system     kube-proxy-sz6rw                             1/1       Running   1          1d
kube-system     kube-scheduler-yoda.me.com            1/1       Running   1          1d
kube-system     kubernetes-dashboard-5bd6f767c7-stscl        1/1       Running   1          23h
kube-system     tiller-deploy-59d854595c-tngg6               1/1       Running   0          23h

I then create my ingress with the following file

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - http:
      paths:
      - path: /whoop
        backend:
          serviceName: test-deployment-service
          servicePort: 80

Then I run kubectl get services --all-namespaces to find my ingress controller port.

NAMESPACE       NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                ClusterIP   10.96.0.1        <none>        443/TCP                      1d
default         my-test                   ClusterIP   10.110.102.5     <none>        80/TCP                       4h
default         test-deployment-service   ClusterIP   10.104.239.37    <none>        80/TCP                       23h
ingress-nginx   default-http-backend      ClusterIP   10.105.218.87    <none>        80/TCP                       29m
ingress-nginx   ingress-nginx             NodePort    10.110.39.104    <none>        80:30652/TCP,443:31624/TCP   27m
kube-system     kube-dns                  ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP                1d
kube-system     kubernetes-dashboard      ClusterIP   10.100.219.110   <none>        443/TCP                      23h
kube-system     tiller-deploy             ClusterIP   10.101.179.202   <none>        44134/TCP                    23h

Now I go to curl my server:

curl -v myserver:30652/whoop/

The resulting response is: 

*   Trying myip...
* TCP_NODELAY set
* Connected to myserver (myip) port 30652 (#0)
> GET /whoop/ HTTP/1.1
> Host: myserver:30652
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Server: nginx/1.13.9
< Date: Fri, 02 Mar 2018 14:38:21 GMT
< Content-Type: text/html
< Content-Length: 187
< Connection: keep-alive
< Location: https://myserver:30652/whoop/
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body bgcolor="white">
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.13.9</center>
</body>
</html>
* Connection #0 to host myserver left intact

I would expect to be getting the response from my pods. However I always get the 308.

If I curl a random path

curl myserver:30652/random

The response I get is 

default backend - 404

How can I hook up my ingress and ingress controller for the routing to work correctly?

3
nginxkubernetesnginx-ingress
when it redirects, what is the Location header of the response (use curl -v). and what happens if you try curl myserver:30652/whoop/ (with a trailing slash)? – Jonah Benton
308 is the default redirect status code for nginx now. the difference between 308 and 301 is that user agents don't preserve the method on 301, and they do on 308. – Jonah Benton
@JonahBenton I tried that and edited the original question. In short the slash doesn't make a difference. – nbroeking
Ok So if 308 is the default redirect status code I guess the question should be why am I getting a redirect instead of my app? – nbroeking
Ah, look at the Location: header- it says https:. The server is telling the browser that it needs to redirect to https from http. There also is a Strict-Transport-Security header- this is an instruction to the browser that it should ALWAYS access this domain (including subdomains, with the additional annotation in the header) with https. I don't believe that's the default in the ingress- so is that coming from your app? – Jonah Benton

3 Answers

8
votes

The key here is to name a host, otherwise the ingress rule applies to the default server, which is configured to redirect HTTP to HTTPS by default.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /whoop
        backend:
          serviceName: test-deployment-service
          servicePort: 80

You should be able to disable the redirect behavior for the default server, bit I haven't tried that.

12
votes

I was having the same problem, but I did not want to add the host value. To prevent the redirect HTTP to HTTPS by the default server like David mentioned, instead I added this annotation to the ingress and it fixed it for me: nginx.ingress.kubernetes.io/ssl-redirect: "false".

My working ingress looks like this now:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "internal-nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: app1
          servicePort: 80
      - path: /app2/*
        backend:
          serviceName: app2
          servicePort: 80
2
votes

I was having issues with HTTP calls getting redirected to HTTPS (verified it with fiddler logs).

Adding this to my yml file made it work for me.

Annotations:

nginx.ingress.kubernetes.io/ssl-redirect: "false"