Kubernetes nginx ingress periodically gives 404

Question

I have deployed kops k8s in AWS, everything in the same namespace.

nginx ingress controller route traffic to https backends (wordpress apps).

I'm able to reach the website, but unfortunately for every 10~ calls only 1 call get http 200. all the other 9 get 404 nginx not found. tried to search everywhere but no luck :(

My configuration: DNS -> AWS NLB -> 2 Nodes

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "True"
    nginx.org/ssl-services: test-service
    nginx.ingress.kubernetes.io/affinity: "cookie"
spec:
  rules:
  - host: "test.example.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: test-service
          servicePort: 8443

nginx-service.yaml:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

nginx-daemonset.yaml:

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: nginx-ingress-controller
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      imagePullSecrets:
      - name: private-repo
      containers:
        - name: nginx-ingress-controller
          image: private_repo/private_image
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --default-ssl-certificate=$(POD_NAMESPACE)/tls-cert
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 33
          resources:
            limits:
              cpu: 500m
              memory: 300Mi
            requests:
              cpu: 400m
              memory: 200Mi
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10

wordpress.yaml:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: test-example
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      restartPolicy: Always
      volumes:
      - name: volume
        persistentVolumeClaim:
           claimName: volume-claim
      imagePullSecrets:
      - name: private-repo

      containers:
      - name: test-example-httpd
        image: private_repo/private_image
        imagePullPolicy: Always
        ports:
        - containerPort: 8443
          name: https

      - name: test-example-php-fpm
        image: private_repo/private_image
        imagePullPolicy: Always
        securityContext:
          runAsUser: 82
        securityContext:
          allowPrivilegeEscalation: false

---
apiVersion: v1
kind: Service
metadata:
  name: test-service
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  ports:
  - name: https-web
    targetPort: 8443
    port: 8443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---UPDATE---

kubectl get  endpoints,services -n example-ns
NAME                           ENDPOINTS                                            AGE
endpoints/ingress-nginx        100.101.0.1:8443,100.100.0.4:443,100.101.0.2:443     1d
endpoints/test-service   100.100.0.1:8443,100.101.0.1:8443,100.101.0.2:8443   4h

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE
service/ingress-nginx        LoadBalancer   SOME-IP     sometext.elb.us-west-3.amazonaws.com   80:31541/TCP,443:31017/TCP   1d
service/test-service   ClusterIP      SOME-IP   <none>                                                                          8443/TCP                     4h

Thanks!

Hi, Can you check the logs of nginx-ingress-contoller daemonset, whether to see if both of these pods are able to find the test-example pods? — Suresh Vishnoi
Hi, Thanks. kubectl logs nginx-pod - 2019/02/20 13:05:36 [error] 136#136: *926 connect() failed (111: Connection refused) while connecting to upstream, client: 100.100.0.0, server: test.example.com, request: "GET / HTTP/2.0", upstream: "100.100.0.2:8443", host: "test.example.com" Just to clarify, apache-pod ip is 100.100.0.1, nginx-pod ip is 100.100.0.2. seems weird that nginx send traffic to 100.100.0.0 — Yonatan.be
Hi, can you run the following command kubectl get endpoints,services — Suresh Vishnoi
Hi, I do not understand why does this endpoint ` 100.101.0.1:8443` exits in both of these endpoints/ingress-nginx and endpoints/test-service — Suresh Vishnoi

Yonatan.be Yonatan.be · Accepted Answer · 2019-02-24T13:10:29

Apparently changing the annotation nginx.ingress.kubernetes.io/ssl-passthrough from "True" to "False" solved it.

Probably has to do something with ssl termination in NGINX and not in the apache.

Kubernetes nginx ingress periodically gives 404

1 Answers