Django - waiting time for a particular view in production takes too long

Question

I've been wrestling with this problem for at least two days. There's a view that produces a page showing all phone calls that a certain user/phone extension made. Nothing fancy, just a long page, maximum of 1000 lines.

This view function receives some parameters from the url in order to choose what to show on this page. There are two situations, one where an "extension=xxxxxx" is passed and one where "user=xxxx" is passed in the url:

if request.GET.get("extension", None):
    # extension_query expects only one Extension object
    extension_query = Extension.objects.filter(number=request.GET["extension"])
    ... # Here I do some conditionals to match the right Extension object.
elif request.GET.get("user", None):
    ... # Simple stuff, nothing to significant.
# at the end I call render_to_response normally

Edit - this is the piece of code that calls my customized render_to_response:
Edit2 - after John C's suggestion of forcing the evaluation of the queryset, the load time decreased from 1.6 min to 14s: Now I cast my calls to a list() before passing it to my customized render_to_response:

if (request.GET.get("format", None) == "screen"
    or request.GET.get("print", False)):
    ctx = dict(calls=list(calls), client=client, extension=extension,
               no_owner_extension=no_owner_extension,
               start=start_date, end=end_date, tab="clients",
               owner=user)
    return finish_request(
        request, "reports/exporting_by_call_type.html", ctx)

This is my customized render_to_response:

def finish_request(request, template, context):
    if "print" in request.GET or "print" in request.POST:
        context.update(printing=True)
    return render_to_response(
        template, RequestContext(request, context))

In my development machine, it takes 3-4 seconds to fully process this view for a realistic data scenario according to the Chrome audit for BOTH situations. In production it takes 3-4 seconds to load the view where user parameter is passed in the URL, but when extension is passed, it takes 2 minutes!

Edit: It's important to say that the difference between passing user or extension in the URL doesn't change the final rendered page except one line at the top where I indicate since when the extension number belongs to someone. The rest of the data is exactly the same.

I profiled every little block in my code involved in generating this view. I timed how long the view takes to render_to_response in my production code (0.05 seconds). I took out parts where extension is called in my template but with no success. I also used django_debug_toolbar to see what every SQL statement is doing and at most, the queries take 2 seconds.

I should also add that I'm using mod_wsgi, debug=False on my production settings... YSlow rates this page a 94 despite taking 2 minutes.

Can anyone shed some light?

Minor note - while browsing through the Django docs, I noticed the repr() Queryset function, which forces evaluation of the queryset, but without turning it into a List. On the off-chance that you (or someone) doesn't want a list, but still wants to force evaluation of the queryset, this is an option. — John C

John C John C · Accepted Answer · 2011-05-18T20:03:58

I don't see anything obviously wrong with that code. One question I have, is are you expecting exactly one object to match? If so, might be worth trying this code:

getData = request.GET.copy() # optional, I like my own copy.
if 'extension' in getData:
    ext = getData['extension']
    extObj = Extension.objects.get(number__exact=ext) # double-underline
# elif...

Note that will result in an Extension object, not a Queryset. If you have multiple extensions, then you do need to use filter - but I'd still suggest using exact, as well as moving the reading of ext from the dictionary, to outside of the filter statement.

Update (from my comments) - try forcing the Queryset to be evaluated immediately using a call to list(), see if that has an effect on render_to_response.

Update 2 - since it did have an effect - here's what I think might be happening. Since Django Querysets use lazy evaluation, when the template finally executes - it's calling an iterator, not an existing list. Maybe there's just too many function calls going on, and each call to the iterator to get a new value, is incurring a lot of unnecessary overhead. Or maybe there's a bug. :)

Django - waiting time for a particular view in production takes too long

1 Answers