Cloud Firestore - Slow performance with looped getDocument request (500+ documents) (Swift)

Question

I'm coding a function that pulls information from an API call, parses the resulting JSON, loops through each result item and then displays this information on the following View Controller. Within this loop, I have to pull information for each item from our Firestore database.

I'm using Swift 4 & Firestore 0.8.0

The code below is what I've been using so far, which takes into account if an item might not exist on our database, with a segue occurring only when all Firestore requests have been completed:

for item in results {
    dispatch.enter()
    //Main loop code for processing the API Call & pulling the document ID for this item

    let docRef = db.collection("collection").document(documentID)
    docRef.getDocument { (document, error) in
        if (document?.exists ?? false), error == nil {
            if let document = document {
                let data = document.data()
                print("EXISTS")
                //do things
                dispatch.leave()
            } else {
                print("Document does not exist")
                dispatch.leave()
            }
        } else {
            print("DOES NOT EXIST") 
            //do other things
            dispatch.leave()
        }
    }
}
dispatch.notify(queue: .main) {
    //perform a segue to the data display VC
}

(The code was written using this question -> Crashes with Firestore, whereby currently Firestore doesn't initially return nil values if documents don't exist in your collection)

The problem I have is that this function ends up taking several minutes to complete. Is there a faster way of performing this looped document request? Is this just beta performance?

Our Firestore collection will eventually have over 1,000,000 documents, and the fields that are pulled from each document are of a nested origin as follows:

collection {
    document {
        object {
           item1: data to pull
           item2: data to pull
           ...etc
        }
    }
}

Any help would be greatly appreciated!

UPDATE & CONCLUSION

Seems like I was taking the reverse approach to this! Since the nature of our database means that we store all documents of the API data, I can perform a single query on our Firestore database to limit data before forcing the API call to only return results based on documents found by the initial query. This removes the need to check if any API call result documents are present in our database! Easy-peasy!

Is there a reason you can't just load the entire collection instead of looping through and loading one document at a time? — Morgan Chen
@MorganChen If you're referring to getDocuments then it would be very performance heavy on a user's device. They'd be loading 1,000,000 documents of 50kb size in order to display maybe 5kb worth of data from each document of around 500-1,000 results — Hendies
I may be misunderstanding, but is there a reason you can't limit your query to only load those 500-1000 results instead of loading all 1000000? — Morgan Chen
the only way we could query our database entries this way would be via id fields collected by our API calls. It depends if a query could be made from an array of ids like this, and a single getDocuments could be performed from the query. The ids collected from the API call results constantly change — Hendies
All sorted! See my question update for details - the problem was with my approach to the API calls themselves — Hendies

Todd Kerpelman Todd Kerpelman · Accepted Answer · 2017-10-13T18:59:32

I think the problem is you're making 500+ separate fetch requests to the database. That's probably going to be slow no matter what.

I would look for ways you could put this data in a collection and then query the collection to get just the documents you need. That would at least allow you to get this data with one call (or, even better, get paginated data if you don't need all 500 documents at once)

Cloud Firestore - Slow performance with looped getDocument request (500+ documents) (Swift)

1 Answers