Flask and Transfer-Encoding: chunked

Question

We're trying get a Flask web service working, and we're having some issues with streaming posts - i.e. when the header includes Transfer-Encoding: chunked.

It seems like the default flask does not support HTTP 1.1. Is there a work around for this?

We are running this command:

$ curl -v -X PUT  --header "Transfer-Encoding: chunked" -d @pylucene-3.6.1-2-src.tar.gz "http://localhost:5000/async-test"

Against this code:

@app.route("/async-test", methods=['PUT'])
def result():
    print '------->'+str(request.headers)+'<------------'
    print '------->'+str(request.data)+'<------------'
    print '------->'+str(request.form)+'<------------'
    return 'OK'

Here's the curl output:

$ curl -v -X PUT  --header "Transfer-Encoding: chunked" -d @pylucene-3.6.1-2-src.tar.gz "http://localhost:5000/async-test"
* About to connect() to localhost port 5000 (#0)
*   Trying ::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5000 (#0)
> PUT /async-test HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: localhost:5000
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 2
< Server: Werkzeug/0.8.3 Python/2.7.1
< Date: Wed, 02 Jan 2013 21:43:24 GMT
<

And here's the Flask server output:

* Running on 0.0.0.0:5000/ 
------->Transfer-Encoding: chunked
 Content-Length:
 User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
 Host: localhost:5000
 Expect: 100-continue
 Accept: */*
 Content-Type: application/x-www-form-urlencoded

 <------------
 -------><------------
 ------->ImmutableMultiDict([])<------------

Answer 1

Its not the Flask Python, its the mod_wsgi. Only mod_wsgi versions 3.0+ started to support chunked http transfers. Flask Python internally use Werkzeug tool-kit as an interface to mod_wsgi. If you installed it from the apt sources it may be an old version.

Try compiling the latest version of mod_wsgi and then install the Flask framework, it may solve the problem.

Answer 2

This works for me but its not the most elegant way of shimming chunked parsing. I used the method of sticking the body into the environment of the response.

Get raw POST body in Python Flask regardless of Content-Type header

But added code to deal with chunked parsing

class WSGICopyBody(object):
    def __init__(self, application):
        self.application = application

    def __call__(self, environ, start_response):
        from cStringIO import StringIO
        input = environ.get('wsgi.input')
        length = environ.get('CONTENT_LENGTH', '0')
        length = 0 if length == '' else int(length)
        body = ''
        if length == 0:
            environ['body_copy'] = ''
            if input is None:
                return
            if environ.get('HTTP_TRANSFER_ENCODING','0') == 'chunked':
                size = int(input.readline(),16)
                while size > 0:
                    body += input.read(size+2)
                    size = int(input.readline(),16)
        else:
            body = environ['wsgi.input'].read(length)
        environ['body_copy'] = body
        environ['wsgi.input'] = StringIO(body)

        # Call the wrapped application
        app_iter = self.application(environ, 
                                    self._sr_callback(start_response))

        # Return modified response
        return app_iter

    def _sr_callback(self, start_response):
        def callback(status, headers, exc_info=None):

            # Call upstream start_response
            start_response(status, headers, exc_info)
        return callback


app.wsgi_app = WSGICopyBody(app.wsgi_app)

use this to get at it

request.environ['body_copy']