Nginx Ingress controller 502 bad gateway for large file uploads

3
votes

We recently upgraded from nginx ingress controller from 0.8.2 to 0.11.0, and started getting 502 bad gateway error on large file uploads around 10 MB or higher, we have set the client_max_body_size to 500m through proxy-body-size in the configmap and verified its set. The smaller files around 5-6 MB works fine.

There are no errors in the logs, just these messages.

redacted - [redacted] - - [25/Mar/2018:02:08:49 +0000] "POST /redacted/upload HTTP/1.1" 000 0 "https://redacted/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 3371263 10.850 [uploader-443] ----

and

[warn] 30684#30684: *42090 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000482, client: redacted, server: redacted, request: "POST /redacted/upload HTTP/1.1", host: "redacted", referrer: "https://redacted/"

The proxied server is tomcat and requests do not make it to tomcat. We have tried increasing:

  • timeouts
  • proxy_buffers
  • proxy_buffer_size

but nothing worked.

Going back to 0.8.2 version resolves the issue.

update 1: nginx.conf snippet

location /redacted/ {

port_in_redirect off;

set $proxy_upstream_name "redacted-443";

....

....

client_max_body_size "500m";

4
kuberneteskubernetes-ingress
Did you read the relevant changelogs? (e.g. github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.9.0) – Oliver Charlesworth
I did, nothing jumps out that would explain the issue. – Zarrar

4 Answers

4
votes

It looks like you may need to set the bigger file size globally for all Ingress rules.

This can be achieved by using proxy-body-size parameter i.e.:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 20mb
....

Afterwards it should work for files bigger than 10MB.

1
votes

Did you verify the generated nginx.conf to see if the value is actually set? I used the in the ingress config itself and it works (version 0.11 as well)

1
votes

I'd recommend using a configmap setting:

  body-size: "1024m"
  proxy-body-size: "1024m"

Here is a full example: https://github.com/mateothegreat/k8-byexamples-ingress-controller/blob/master/manifests/configmap.yaml

To apply a configmap to the ingress-controller deployment you'll want to pass it as an argument to the container such as: https://github.com/mateothegreat/k8-byexamples-ingress-controller/blob/master/manifests/controller-deployment.yaml#L59

0
votes

I was facing the same problem. In our case, it was a python Django application running with uwsgi with Nginx setup as ingress controller. When a file greater than 50mb was being uploaded I would get 502, despite the fact that I had added all the configuration to the ingress controller config map. It turned out that the issue was with the pod's liveliness probe. When a large file is sent to the application the uwsgi server that is running as a single thread, get busy with that request and the liveliness checks time out. Therefore the pod gets restarted. I tested this by removing the liveliness check from the deployment and the file was uploaded without any problem.