How to create index for django sitemaps for over 50.000 urls

5
votes

I have the following url configuration 

url(r'^sitemap\.xml$', index, {'sitemaps': sitemaps}),
url(r'^sitemap-(?P<section>.+)\.xml', cache_page(86400)(sitemap), {'sitemaps': sitemaps}),

and sitemaps include following sitemap

 class ArticlesDetailSiteMap(Sitemap):
    changefreq = "daily"
    priority = 0.9

    def items(self):
        return Article.objects.filter(is_visible=True, date_published__lte=timezone.now())

but there are more than 50.000 articles. So i get timeout error when i try /sitemap-articles.xml because it tries to get all the articles.

Any ideas how should i create an index and make the pagination work here as it says in the documentation below,

https://docs.djangoproject.com/en/dev/ref/contrib/sitemaps/#creating-a-sitemap-index

2
pythondjangositemapdjango-sitemaps
Did you figure out how to do it in the end? - ip.

2 Answers

3
votes

Try this

from django.core.paginator import Paginator, PageNotAnInteger, EmptyPage

And then

article_list = Article.objects.filter(is_visible=True, date_published__lte=timezone.now())
paginator = Paginator(article_list, 10)
page = request.GET.get('page')


try:
    articles = paginator.page(page)
except PageNotAnInteger:
    articles = paginator.page(1)
except EmptyPage:
    articles = paginator.page(paginator.num_pages)

And you can access the site map using the URLs like sitemap\.xml?page=5

2
votes

I have put limit=5000 and issue resolved. 

class ArticlesDetailSiteMap(Sitemap):
    changefreq = "daily"
    priority = 0.9
    limit = 5000

    def items(self):
        return Article.objects.filter(is_visible=True, date_published__lte=timezone.now())

and it created paginated urls for all Articles paginated by 5000