How to query thousands of values and only return ones in array - Firebase Swift

Question

In firebase realtime database, I have data as such:

"users" : {
"37KfZKDwrieIEKI9juAC4Xm8aPi1" : {
  "isConnected" : false,
  "isGuestUser" : false,
  "lastOnline" : 1510250272022,
  "profilePicture" : "5a039030515e78653148",
  "userID" : "37KfZKDwrieIEKI9juAC4Xm8aPi1",
  "username" : "adsfasdfasdf"
},
"4D1GNiRH5NeRxpmTNg6JhJ3iTck1" : {
  "isConnected" : false,
  "isGuestUser" : true,
  "lastOnline" : 1510077502788,
  "profilePicture" : "5a01f2648278b6652011",
  "userID" : "4D1GNiRH5NeRxpmTNg6JhJ3iTck1",
  "username" : "ihoho"
},
"5UA3INZ7i0dnNtgX0ai5ABhjxh43" : {
  "isConnected" : false,
  "isGuestUser" : true,
  "lastOnline" : 1512610102474,
  "profilePicture" : "5a14df775a34f2388873",
  "userID" : "5UA3INZ7i0dnNtgX0ai5ABhjxh43",
  "username" : "jlkjlkjlkj"
},...

I am using an external API that returns a Json that looks like this

"candidates" : [
    {
      "enrollment_timestamp" : "1510182689539",
      "subject_id" : "37KfZKDwrieIEKI9juAC4Xm8aPi1",
    },
    {
      "enrollment_timestamp" : "1513557650425",
      "subject_id" : "CKUVZ7XtY9VKJakn1lBV7MVW1702",
    },
    {
      "enrollment_timestamp" : "1507578748901",
      "subject_id" : "l7VDdtGFpMe8BRbrlCyAciTvONk1",
    },...

The ultimate goal is to get all the users from the json from the external api that are online. This requires listening to the 'isConnected' endpoint in each user, and determining if it is true or false.

Now this is impossible using firebase and my current data structure, because firstly firebase does not support multiple query parameters, so I cannot do where userID = subjectID && where isConnected == true, and secondly, more importantly, firebase does not let me do the equivalent of WHERE IN, i.e. mapping userID's to an array of potential userIDs supplied client side (fyi subjectID === userID)

So what I did was restructure my data like so;

"onlineUsers" : {
"Aze1x7jTZIbPyyPmlUYWVdEyAd73" : true,
"CQdmMxkmqBerRGOQpFblx7SO4D33" : true,
"EptMK62Kt1Sp1EIb5meuKHVpUqs1" : true,
"J2X65KauDlSvkN4Yp5V4IF0sTnx1" : true,
"KnYEqsekY9YXV3ayOx082xw8VQX2" : true,
"aGLrKH31YvRKrB8KYQmZ4sA122m1" : true,
"ab3JZyw9YMdpW3mXkZc2BjYkxej2" : true,
"gpQic1EzSnXL9x5DhaoXxcWrGF22" : true,
"qQaBPMaDOrXrWddsijfMJWusuGG3" : true,
"tDWEUoKS4mUdQ1bWeiLTlhwCSoD3" : true

},

Now all I have to do to get all online users that are in the external api json is query onlineUsers with the condition parameter checking by the key. That way I know if the result is null, the user is not online:

    static func queryDatabase(child: String, queryEqual: String, keyOf: Int, completionHandler: @escaping (_ return: AnyObject?, _ error: String?) -> Void){
    print("querying db with child ", child)

    let ref = databaseReference.child(child).queryOrderedByKey().queryEqual(toValue: queryEqual)

     ref.observe(.value, with:{ (snapshot: DataSnapshot) in
        print("subjectID: ", queryEqual, "at key ", keyOf)
        print("queryResult", snapshot)

        if let value =  (snapshot.value as? [String: AnyObject])?[queryEqual] {
            print("unwrapped snapshot dict value from key: ", value)

            completionHandler(value,  nil)

        }else{
            print("no value for key \(queryEqual) so setting return as nil")

            completionHandler(nil,  nil)

        }



    }){ (error) in
        print(error.localizedDescription)

        completionHandler(nil,  error.localizedDescription )


    }

}

This function would be called simple by looping through the external api json, and calling this function ^^ every iteration, something like:

      for (key, valueSubjectID) in arrayOfOrderedMatches.enumerated(){
         //call function queryDatabase here

    queryDatabase(child: "onlineUsers", queryEqual: valueSubjectID, keyOf: key, completionHandler:{ (response, error) in


//it would return the user if they were online here


)}
    }

Now. This works. It is the most efficient way I can possible think of doing it after looking around the internet and considering every possibility. However, now onto the big problem: the json from the api can be thousands and thousands of users long. This means thousands of listeners would be being attached to check each individual user to see if they are online. This, as I have been told by a Firebase developer, is not ideal. I should not be attaching thousands of listeners. Additionally, for some reason, listeners stop being added at around 3500, i'm guessing because it bugs out. I can't remove the listeners if the query returns null (ie offline) because of the way the cache offline persistence works, and even if I could I don't think this would solve the problem. Is there any way this problem can be solved?

Answer 1

Let me restate the objective.

The app is provided a large list of user id's (as JSON) and the goal is to know which of those users are online.

Here's two options:

We start with a users node

users
  uid_0
    name: "Frank"
    online: true
  uid_1
    name: "Jeff"
    online: false

When the JSON of the users we want to check is received, we convert it to an Array for easy access.

Then, we have a couple of options

1) Iterate over the array and observeSingleEvent for each uid to get it's current status. This won't be a query since we can directly access the uid so it will be fairly lightweight by comparison to a query.*

let arrayToTest = [String]() //the array of uid's to test
for uid in arrayToTest {
    let thisUserRef = self.ref.child("users").child(uid)
    thisUserRef.observeSingleEvent(of: .value, with: { snapshot in
        let dict = snapshot.value as! [String: Any]
        let status = dict["online"] as! Bool
        print("uid: \(uid) online status is: \(status)")
    })
}

2) Let Firebase do the heavy lifting by adding a .childAdded event to the users node, which will iterate over all of the users one at a time.

Compare the uid in the snapshot to the uid's in the array and if it matches, get it's status (on/offline) and if it doesn't exist in the array, ignore it.

let usersRef = self.ref.child("users")
let arrayToTest = [String]() //the array of uid's to test
usersRef.observe(.childAdded, with: { snapshot in
    let dict = snapshot.value as! [String: Any]
    let uid = dict["user_id"] as! String
    if arrayToTest.contains(uid) {
        let status = dict["online"] as! Bool
        print("uid: \(uid) online status is: \(status)")
    }
})

Option 2) will generally be better as it's only loading one at a time and Firebase is doing all of the work - we're just checking to see if the uid from the snapshot exists in code.

3)

As a third option we can leverage a query. Note that in the first two options we retrieved our data using the lightweight observe function. Queries are heavier by comparison and utilize more resources - however, the snapshot results are limited to only those users that are logged in, and if there are a significant amount of users may be the best option.

let usersRef = self.ref.child("users")
let queryRef = usersRef.queryOrdered(byChild: "online").queryEqual(toValue: true)
let arrayToTest = [String]() //the array of uid's to test
queryRef.observe(.childAdded, with: { snapshot in
    let dict = snapshot.value as! [String: Any]
    let uid = dict["user_id"] as! String
    if arrayToTest.contains(uid) {
        print("uid: \(uid) online status is: true")
    }
})

*Firebaser's will frown on 1) as they strongly recommend against doing observes in a tight loop. So 2) if you want to get confirm every user exists and 3) if you're just interested in online users.