How to batch more than 500 operations in python for firestore?

Question

I'm creating documents from web scraping in python and uploading them to Firestore. To do so I'm adding them to a dictionary and uploading them from a for loop in python, one by one(ideally would be better to upload the collection at once, but that doesn't seem an option). I want to use batches, however they have the 500 limit per batch and I need to do more than 100,000 operations. The operations are merely set() operations and a couple of update() Is there a function to know the current size of the batch so I can reinitialize it? What is the best way to use batches for more than 500 operations in python?

Guanaco Devs Guanaco Devs · Accepted Answer · 2019-01-17T23:29:25

The best way I found to deal with the 500 batch limit, while working with python, is to put all my data I want to send to Firestore in a 'Flat' dictionary so I can work with every unique document. This dictionary has as key for every document the form of: 'collection_document_collection_document...' while the value for that key would be a dictionary with:

{'action': 'set', 'reference': reference, 'document': {}}

The 'action' can be 'set', 'update' or 'delete', the 'reference' key is an actual Firestore reference, and the 'document' is just the document. For instance this are 2 documents in different locations.

{
    'user_data_roger':
    {'action': 'set', 'reference': db.collection('user_data').document('roger'), 'document': {'name': 'Roger', 'age': 37}},
    'user_data_roger_works_april':
    {'action': 'update', 'reference': db.collection('user_data').document('roger').collection('works').document('april'), 'document': {'is_valid': True, 'in_progress': True, 'level':5}},
}

After processing all the data I need I want to split the dictionary in arrays of 500 items and then add all of those Items to the batch, using the 'action' key for the batch.

# Convert dictionary to a list
dictionary_list = []
for item in dictionary:
    dictionary_list.append(dictionary.get(item))
# Split List in lists containing 500 items per list
list_to_batch = [dictionary_list[item:item+500] for item in range(0, len(dictionary_list), 500)]
# Finally iterate through the 'list_to_batch' add each item to the batch and commit using a for loop
for item in list_to_batch:
    batch = db.batch()
    for document in item:
        if document['action'] == 'set':
            batch.set(document['reference'], document['value'])
        elif draw['action'] == 'update':
            batch.update(document['reference'], document['value'])
        else:
            batch.delete(document['reference'], document['value'])
    # Finally commit the batch
    batch.commit()

In my particular case after processing all the data I needed, I ended up with more than 700,000 operations, so watch out for the billing :-D

How to batch more than 500 operations in python for firestore?

2 Answers