Using firebase cloud function to copy all documents from a master collection in Firestore to new sub-collection

Question

I have a master collection in firestore with a couple hundred documents (which will grow to a few thousand in a couple of months).

I have a use case, where every time a new user document is created in /users/ collection, I want all the documents from the master to be copied over to /users/{userId}/.

To achieve this, I have created a firebase cloud function as below:

// setup for new user
exports.setupCollectionForUser = functions.firestore
  .document('users/{userId}')
  .onCreate((snap, context) => {

    const userId = context.params.userId;

    db.collection('master').get().then(snapshot => {

      if (snapshot.empty) {
        console.log('no docs found');
        return;
      }

      snapshot.forEach(function(doc) {

        return db.collection('users').doc(userId).collection('slave').doc(doc.get('uid')).set(doc.data());
      });
    });
 });

This works, the only problem is, it takes forever (~3-5 mins) for only about 200 documents. This has been such a bummer because a lot depends on how fast these documents get copied over. I was hoping this to be not more than a few seconds at max. Also, the documents show up altogether and not as they are written, or at least they seem that way.

Am I doing anything wrong? Why should it take so long?

Is there a way I can break this operation into multiple reads and writes so that I can guarantee a minimum documents in a few seconds and not wait until all of them are copied over?

Please advise.

Renaud Tarnec Renaud Tarnec · Accepted Answer · 2020-02-07T15:11:17

If I am not mistaking, by correctly managing the parallel writes with Promise.all() and returning the Promises chain it should normally improve the speed.

Try to adapt your code as follows:

exports.setupCollectionForUser = functions.firestore
    .document('users/{userId}')
    .onCreate((snap, context) => {

        const userId = context.params.userId;

        return db.collection('master').get().then(snapshot => {  

            if (snapshot.empty) {
                console.log('no docs found');
                return null;
            } else {

                const promises = [];
                const slaveRef = db.collection('users').doc(userId).collection('slave');
                snapshot.forEach(doc => {

                      promises.push(slaveRef.doc(doc.get('uid')).set(doc.data()))

                });

                return Promise.all(promises);

            }

        });

    });

I would suggest you watch the 3 videos about "JavaScript Promises" from the Firebase video series which explain why it is key to return a Promise or a value in a background triggered Cloud Function.

Note also, that if you are sure you have less than 500 documents to save in the slave collection, you could use a batched write. (You could use it for more than 500 docs but then you would have to manage different batches of batched write...)

Using firebase cloud function to copy all documents from a master collection in Firestore to new sub-collection

1 Answers