I have a middleware where I a want to log every request/response. How can I access to POST data?

Question

I have this middleware

import logging

request_logger = logging.getLogger('api.request.logger')


class LoggingMiddleware(object):

    def process_response(self, request, response):
        request_logger.log(logging.DEBUG,
               "GET: {}. POST: {} response code: {}. response "
               "content: {}".format(request.GET, request.DATA,
                                       response.status_code,
                                       response.content))
        return response

The problem is that request in process_response method has no .POST nor .DATA nor .body. I am using django-rest-framework and my requests has Content-Type: application/json

Note, that if I put logging to process_request method - it has .body and everything I need. However, I need both request and response in a single log entry.

Answer 1

Here is complete solution I made

"""
Api middleware module
"""
import logging

request_logger = logging.getLogger('api.request.logger')


class LoggingMiddleware(object):
    """
    Provides full logging of requests and responses
    """
    _initial_http_body = None

    def process_request(self, request):
        self._initial_http_body = request.body # this requires because for some reasons there is no way to access request.body in the 'process_response' method.


    def process_response(self, request, response):
        """
        Adding request and response logging
        """
        if request.path.startswith('/api/') and \
                (request.method == "POST" and
                         request.META.get('CONTENT_TYPE') == 'application/json'
                 or request.method == "GET"):
            request_logger.log(logging.DEBUG,
                               "GET: {}. body: {} response code: {}. "
                               "response "
                               "content: {}"
                               .format(request.GET, self._initial_http_body,
                                       response.status_code,
                                       response.content), extra={
                    'tags': {
                        'url': request.build_absolute_uri()
                    }
                })
        return response

Note, this

'tags': {
    'url': request.build_absolute_uri()
}

will allow you to filter by url in sentry.

Answer 2

Andrey's solution will break on concurrent requests. You'd need to store the body somewhere in the request scope and fetch it in the process_response().

class RequestLoggerMiddleware(object):

    def process_request(self, request):
        request._body_to_log = request.body

    def process_response(self, request, response):
        if not hasattr(request, '_body_to_log'):
            return response

        msg = "method=%s path=%s status=%s request.body=%s response.body=%s"
        args = (request.method,
                request.path,
                response.status_code,
                request._body_to_log,
                response.content)

        request_logger.info(msg, *args)

        return response

Answer 3

All answers above have one potential problem -- big request.body passed to the server. In Django request.body is a property. (from framework)

@property
def body(self):
    if not hasattr(self, '_body'):
        if self._read_started:
            raise RawPostDataException("You cannot access body after reading from request's data stream")
        try:
            self._body = self.read()
        except IOError as e:
            six.reraise(UnreadablePostError, UnreadablePostError(*e.args), sys.exc_info()[2])
        self._stream = BytesIO(self._body)
    return self._body

Django framework access body directly only in one case. (from framework)

elif self.META.get('CONTENT_TYPE', '').startswith('application/x-www-form-urlencoded'):

As you can see, property body read the entire request into memory. As a result, your server can simply crash. Moreover, it becomes vulnerable to DoS attack. In this case I would suggest using another method of HttpRequest class. (from framework)

def readlines(self):
    return list(iter(self))

So, you no longer need to do this

def process_request(self, request):
    request._body_to_log = request.body

you can simply do:

def process_response(self, request, response):

    msg = "method=%s path=%s status=%s request.body=%s response.body=%s"
    args = (request.method,
            request.path,
            response.status_code,
            request.readlines(),
            response.content)

    request_logger.info(msg, *args)

    return response

EDIT: this approach with request.readlines() has problems. Sometimes it does not log anything.

Answer 4

It's frustrating and surprising that there is no easy-to-use request logging package in Django.

So I created one myself. Check it out: https://github.com/rhumbixsf/django-request-logging.git

Uses the logging system so it is easy to configure. This is what you get with DEBUG level:

GET/POST request url
POST BODY if any
GET/POST request url - response code
Response body

Answer 5

It is like accessing the form data to create a new form.

You must use request.POST for this (perhaps request.FILES is something you'd log as well).

class LoggingMiddleware(object):

    def process_response(self, request, response):
        request_logger.log(logging.DEBUG,
               "GET: {}. POST: {} response code: {}. response "
               "content: {}".format(request.GET, request.POST,
                                       response.status_code,
                                       response.content))
        return response

See Here for request properties.

Answer 6

Also note, that response.content returns bytestring and not unicode string so if you need to print unicode, you need to call response.content.decode("utf-8").

Answer 7

You cannot access request.POST (or equivalently request.body) in the process_response part of the middleware. Here is a ticket raising the issue. Though you can have it in the process_request part. The previous answers give a class-based middleware. Django 2.0+ and 3.0+ allow function based middlewares.

from .models import RequestData # Model that stores all the request data
def requestMiddleware(get_response):
    # One-time configuration and initialization.

    def middleware(request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.
        try : metadata = request.META ;
        except : metadata = 'no data'
        try : data = request.body ;
        except : data = 'no data'
        try : u = str(request.user)
        except : u = 'nouser'

        response = get_response(request)
        w = RequestData.objects.create(userdata=u, metadata=metadata,data=data )
        w.save()
        return response
    return middleware

Model RequestData looks as follows -

class RequestData(models.Model):
    time = models.DateTimeField(auto_now_add=True)
    userdata = models.CharField(max_length=10000, default=' ')
    data = models.CharField(max_length=20000, default=' ')  
    metadata = models.CharField(max_length=20000, default=' ')