Sitemap Generation Solution for a dynamic website

Question

CONTEXT
Assume my website contains a listing of shops organized across country, state, city and locality. Each shop has a dynamically generated web page. The total number of shops would eventually reach ~1.5 million. I use NDB to store shop data. I plan to use XML sitemap and submit them manually to the search engines. I use GAE Python.

PROBLEM
I want to maintain (generate and keep updated) url links in the sitemap for all shop pages. Each unique url link for a shop page contains the following: Country, State, City, Locality, Shop Name, Unique Index

eg, wwww.example.com/country--state--city--locality--shop_name--unique_index

Shops could be added, deleted or their data(eg, their name or city, etc) could be changed. I need to design a solution which helps me have the latest sitemap with updated links for all the shops. I intend to submit a new sitemap as soon as possible after any shop has been added/deleted/updated.

My Approaches
Approach 1
Generate the sitemap on the fly by querying the information from NDB models.
Cons of Approach 1

1. NDB fetch limit of 10,000.
2. Read operation free quota of 50,000.
3. High consumtion of Frontend instance hours.
4. Operation completion time of 60 seconds.

Approach 2
Generate and store the sitemap on my laptop using a program(say X, written by me in Perl/Python). Whenever a shop gets added / deleted / updated on my website, I would update a GCS(Google Cloud Storage) stored file with mnemonics like:

ADD < shop data like name, etc >  
DELETE < shop data like name, etc >  
UPDATE < shop data like name, etc >

I would download and feed this file to my local program X to generate the sitemap by updating the older stored sitemap file.
Cons of Approach 2

• GCS does not allow append of data to a file. Entire file needs to be written every time. So, with increase in the number of shops from 0 to 1.5 million, RAM usage and frontend instance hours consumtipn would peak.
• Operation completion time of 60 seconds.

Approach 3
The sitemap.xml file would contain:

• Entries with URLs for other Sitemap Index files on Country basis. These country sitemap files would contain entries for URLs of State sitemap files. Similarly, state sitemap files would contain entries for URLs of City sitemap files. Similarly, city sitemap file would contain entries for URLs of locality sitemap files. Locality sitemap files would contain entries for URLs of shop pages.

• Entries with URLs for all the static pages(like, FAQ, About Us, etc).

Pros of Approach 3

• When a shop page gets added / deleted / updated, I need to update that particular sitemap file.

Doubts with Approach 3

• Can I store all the sitemap.xml files in GCS? Do you foresee any problem with that?

• Is it allowed to have multiple levels of sitemap index files pointing to other sitemap index files?

I am not able to find a good solution. I have seen similar questions on SO and Nick's blog but in vain. I wish to remain within the free quota if possible. Please provide your suggestions.

Answer 1

I have used the following solution in the past:

• When a url needs to be indexed queue a task with the desired url.
• When the task is processed, save a sitemap entry entity in the datastore with the url (and any other metadata you need)
• A cron job will trigger generation of sitemap files by iterating the sitemap entities, batching to around 50K entries (tune for optimal outcome). These are stored as a file (blobstore, GCS up to you) which can be served directly and referenced by a sitemap file datastore entity.
• Either on request, or using a cron job, you can generate the sitemap index, which includes each sitemap file entity.

This solution is premised on a few caveats:

• Google only checks your sitemap occasionally. If you're getting hammered by google, bing, crawlers etc, you may want to store the sitemap index or cache it.
• Changes to the sitemap don't need to be transactional/current because it isn't used in that manner. For example urls that 404, 301, 302 etc are ok to have around for some amount of time, as are duplicates.
• You can figure out a mechanism on how to remove or expire old sitemap file entities. You could use a TTL, or just never expire them until a full rebuild run every few months (which is very practical).

There are a couple of positives

• You can queue a task to remove a url (i.e delete the datastore entry)
• You can create an admin endpoint to purge everything and regenerate based on the datastore state. This can be very useful when testing with webmaster tools or you change your url generation strategy etc.
• You can create an admin page/endpoint to add files to the sitemap manually (or just push to the queue)
• You can bundle in urls which are not dynamic in the last step, for example privacy policy, landing pages etc

As mentioned, there are limitations on sitemap files which mean you will want to decompose to many sitemap files and a sitemap index. These limitations are roughly max 50K entries per file and 10mb a file. Given you're probably going to be crunching a lot of datastore entries, I would optimise for generating the files (i.e. processing time and optimal batch/chunk sizes) over worrying about filling up each index file to its limits.

Answer 2

You can shard the sitemaps, as you'll have to do anyway if they exceed 50,000 URLs per Sitemap formats and guidelines, and reduce the amount of rewriting you'll have to do for Approach 2, having to just update, say the "country--state--city" section and the Sitemap index file.