Efficient way to query posts of multiple users from a node in Firebase

Question

I am building a social application which allows users to follow each other and get a feed of posts made by users they follow. Currently, am planning my data to be stored as given below, however, am not sure if it will scale.

{
  "users": {
    "alovelace": {
      "name": "Ada Lovelace",
      "follows": { 

                   "ghopper": true,
                   "eclarke": true,
                   "john": true
                   //+100 more maybe
      },
    },
    "ghopper": { ... },
    "john": { ... },
    "eclarke": { ... }

    //all the users....
  },

  "posts": {

      "p1": {
        "user": "eclarke",
        "post": "Its a wonderful day today.",
        "timestamp": 1459361875337
      },
      "p2": {
        "user": "john",
        "post": "The laptop seems to be malfunctioning.",
        "timestamp": 9454361875332
      },
      "p3": {
        "user": "ghopper",
        "post": "The bus is late.",
        "timestamp": 6593618235336
      },
      "p4": {
        "user": "someotheuser",
        "post": "The relay seems to be malfunctioning.",
        "timestamp": 1459361875337
      },
      "p5": {
        "user": "eclarke",
        "post": "The relay seems to be malfunctioning.",
        "timestamp": 5459361875331
      },
      "p6": {
        "user": "alovelace",
        "post": "The relay seems to be malfunctioning.",
        "timestamp": 3459361875334
      }
      //1,000,000+ posts maybe present. 
  }
}

All the users are stored in users node along with whom they follow. And all the posts made by all users are stored in the node posts. While users of the platform may scale to 1000's the no. of posts should easily scale to 1,000,000+. I will need query the data in the following ways:

For the user, alovelace get the recent posts of the users the users follow. In our case, posts of ghopper,eclarke and john. This has to be sorted by time and paginated. As we may only show 10 posts at a time.
For the user `john', show all his posts on his profile.

Is firebase the right DB for this sort of use case. Is the structure am following scalable? Is there anything else I have to consider?

theblindprophet theblindprophet · Accepted Answer · 2017-06-11T13:21:22

I just tackled this recently, you have to use a technique called fanning. It involves replicating some data, but Firebase specifically states in some docs that data is cheaper than time so its okay to replicate if its advantageous.

Create a timeline node in your database as such:

{
    123456: {
        user: "Ada Lovelace",
        posts: {
            32gb2gjkb2g: {
                "user": "someotheuser",
                "post": "The relay seems to be malfunctioning.",
                "timestamp": 1459361875337
            },
            23g23gg2g2: {
                "user": "someotheuser",
                "post": "The relay seems to be malfunctioning.",
                "timestamp": 1459361875337
            },
            ...
        }
    }
}

where 123456 is Ada lovelace's uid and 32gb2gjkb2g and 23g23gg2g2 are the uids of the users who posted.

Every time a user creates a post, get all their followers:

"ghopper": { ... },
"john": { ... },
"eclarke": { ... }

When a user posts, loop through each follower and fan them out into an object:

this.fanoutObj= {};
followers.forEach((follower_uid) => {
    this.fanoutObj['/timelines/' + follower_uid + '/' + some_post_id] = some_post_obj;
});
this.AF.database.object("/").update(this.fanoutObj);

Now you can query a user's following's posts by just querying their timeline, using their uid.

To get of all john's posts, just query the posts node specifying his name.

fanout

Efficient way to query posts of multiple users from a node in Firebase

1 Answers