Firestore - replace collection documents while deleting if needed

Question

Here's a reduced description of a problem I have with Firestore:

I have item documents in a collection.

ItemsCollection
    ItemDoc1
        someField1: someValue
        someField2: someValue
    ItemDoc2
        someField1: someValue
        someField2: someValue
    ItemDoc3
        someField1: someValue
        someField2: someValue

Now, I get an updated list of items from some API that I poll every few seconds, and on every poll I want to write the updated state of items to Firestore.

For example, the initial API call returns

[
    ItemDoc1
        someField1: someValue
        someField2: someValue
    ItemDoc2
        someField1: someValue
        someField2: someValue
    ItemDoc3
        someField1: someValue
        someField2: someValue
]

So when it's written to Firestore, the state at Firestore is the one I wrote above.

Now, after a few seconds, a new API call is made and the response is

[
    ItemDoc1
        someField1: someValue
        someField2: someValue
    ItemDoc2
        someField1: *differentValue*
        someField2: someValue
]

What I need is that when writing to Firestore, the following will happen:

ItemDoc1 will not be written at all because it didn't change
ItemDoc2 will be identified as changed and will be updated
ItemDoc3 will be deleted because it's missing in the new response

I couldn't find a nice way to write this kind of code.

What I tried so far is this: I created a batch and for each item in the API response, I created a doc ref, and add a set command to the batch. Actually, since batches are limited to 500 items and I might have more than that, I split the docs to chunks of 500 items, then I create a separate batch for each chunk, then I run all the batches in parallel. The result - items that were not changed are not rewritten, and items that changed are updated, as expected. However, missing items (ItemDoc3) are not deleted, because I only did set commands so far. In order to identify deletions, I have to keep the current item ids separately, and every time I get a new API response, I have to calculate myself what was removed, and call delete.

This is all VERY cumbersome. Is there some nice way to set a whole collection, so that I can get the behaviour I need out of the box? I thought it would be trivial in Firestore

Frank van Puffelen Frank van Puffelen · Accepted Answer · 2019-08-25T14:22:50

There is no Firestore API to replace an entire collection with new contents. While you could file a feature request for it, I doubt it'll be added anytime soon as it is a quite uncommon API request in my experience.

I see two common approaches to implementing your use-case:

Wipe the entire existing contents and then write all data you got from the API. This will be relatively simple in code, but means that the database will be empty between the two steps. It is possible you don't want such an empty state, since realtime listeners will likely see/show it.
Load the existing contents, loop over that and build an internal state of all the updates you need to perform. When done with the loop, send the updates to Firestore as a set of batched writes or transactions.

It sounds like you're already doing #2. If you're having trouble making this work and can't use #1 instead, we're likely better able to help if you show the minimal code that reproduces the problem.

Firestore - replace collection documents while deleting if needed

1 Answers