Django ListView painfully slow using Haystack

2
votes

How do you speed up a Haystack-search powered (using the Whoosh backend) paginated Django list view?

I had a simple ListView like:

class PersonListView(ListView)
    template_name = 'person-list.html'
    paginated_by = 10
    def get_queryset(self):
        return Person.objects.all()

Returning a page with 3000 results runs in about 1 second on my localhost.

I then "plugged in" Haystack to allow full-text searching on names by doing:

class PersonListView(ListView)
    template_name = 'person-list.html'
    paginated_by = 10
    def get_queryset(self):
        #return Person.objects.all()
        return SearchQuerySet().models(models.Person)

And I setup the appropriate index and run manage.py rebuild_index:

class PersonIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    nickname = indexes.CharField()
    first_name = indexes.CharField()
    middle_name = indexes.CharField()
    last_name = indexes.CharField()

    def get_model(self):
        return models.Person

    def index_queryset(self, using=None):
        return self.get_model().objects.all()

However, after this, the same page now takes about 15 seconds to run...

I tried using a django profiler, but I don't see much difference in the query times before and after the change, and the longest running query takes only a second, indicating there's some bug on the view side that's causing all the query results to be iterated over.

What's causing Haystack to run horribly slow?

2
pythondjangodjango-haystack

2 Answers

2
votes

One reason for you search being slow is, that for every search result haystack will try to get the corresponding object from the database. You can prevent this behaviour by using load_all() on the SearchQuerySet. Then haystack will try to collect all results in one query. Just be careful when displaying stuff coming from related models, as this will cause additional database lookups without any additional configuration.

If you want to avoid database lookups in general and only display data that is indexed (you can also try this to make sure that the slowness is caused by addional database lookups). Otherwise if you need additional debugging, use the debug-toolbar, which will display all database queries and there's also an additional haystack panel for it.

1
votes

There are two things you can do, according to this Haystack issue.

The first one, you can set INCLUDE_SPELLING to False in the settings, as in:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
        'PATH': os.path.join(PATH_TMP, "whoosh_index", "default"),
        'STORAGE': 'file',
        'POST_LIMIT': 128 * 1024 * 1024,
        'INCLUDE_SPELLING': False,
        'BATCH_SIZE': 100,
    },
}

And another thing is that you can set HAYSTACK_ITERATOR_LOAD_PER_QUERY to larger number, for example, 100 instead of the default 10.

HAYSTACK_ITERATOR_LOAD_PER_QUERY = 100