How many documents can be written in batch mode from firebase cloud function trigger to Firestore?

1
votes

We have large amount of data where we need to update documents based on a status. We will writing in batch of 500 chunks and wondering how many max records we can commit with a single trigger invocation?

Our trigger is a pubsub trigger in firebase cloud functions.

We see there is a limit of 540secs per invocation so would like to know how many max documents we can write in batches?

Update : Adding usecase

I have an event collection(Events) where users can subscribe for each event happening in a country.

Users have an api to see how many events they have subscribed to. They have query flags like is the event Live/Finished/Upcoming.

As I can't save list of user array who subscribed for an event in the event document(assuming subscribers can go beyond the document limit when stored), I maintained a separate sub-collection under users collection. Ex : users/user-id/subscribedevents

Event document's status (Live/Finished/Upcoming), i'm updating from a cron job which will be running every minute. This is because I can't apply filters with two different fields (startDate & endDate).

When ever an event's status changes, i need to update in subscribedevents subcollection (which is under user's collection).

As I will be updating all the subscribedevents subcollection entries, I want to do it in batches.

Hope the usecase gives some clarity on where it is applied. As firestore is designed for scale, wondering how others are handling this scenario as its very common.

1
google-cloud-firestoregoogle-cloud-functions
The documented limits for Cloud Firestore don't change when performed from Cloud Functions as compared to mobile, web, or server environments. firebase.google.com/docs/firestore/… – Doug Stevenson
Maximum write rate to a document 1 per sec => Does this mean for 540sec, we can write only 540 documents? Is it the same case when using batches (with each batch containing 500 doc changes)? – Ayyappa
That limit refers to the rate at which you can write to a single document repeatedly, not a batch of documents. – Doug Stevenson
Thanks for clarifying Doug! Do you have any numbers on how many batches possible in 540secs? Actually we couldn't save an array list in one document as its limited to 1mb. so the array list entries are moved to a separate collection but need to handle the changes in the original document. For this we are writing a database trigger to update all the subcollection entries. This subcollection can be nearly million documents. I see this can be an usual scenario when database has higher volume. – Ayyappa
If you're concerned about the amount of time it takes to perform all these batches, maybe Cloud Functions isn't the right way to go about it. Or consider breaking up your batches into known sizes and split all that work up among several function invocations. – Doug Stevenson

1 Answers

-1
votes

Each transaction or batch of writes can write to a maximum of 500 documents. And 20 maximum batched writes.

References

Firestore Write Documents

Transaction and batches in firestore