Create feed like api method with a cursor in Cloud Endpoints (google app engine)

Question

What is a way to create an Api method in objectify that lists all the most recent posts by the people you follow?

Suppose you have Profile entities that write Post entities. Each Post has a Profile as its parent. And each Profile has a list of keys of other Profiles that it follows.

How can I create a query (with a cursor) that lists all the Post's by my followers sorted by date descending?

Should I structure my model differently from what I have described above in order to be able to do this? I thought I could do it this way, but I learned from that post that Objectify has some limitations. One of which being you cannot use a cursor on a IN filter. Documentation here also says:

You can filter on id properties if this query is restricted to a Class and the entity has no @Parent. If you are having trouble working around this limitation, please consult the objectify-appengine google group.

The objectify-appengine google group is not very active...

Answer 1

There are two general approaches to building a feed: Scatter-gather, or preindex. They each have their tradeoffs.

Scatter-gather is usually the easiest to start, especially when you can do joins in the DB. And it's conveniently flexible - change the code and everyone sees the result immediately. However, scalability of this is severely limited, especially if your users typically follow large numbers of others. As you've noticed, it's a PITA in GAE because there are no joins (and an IN only lets you specify a few tens of items). Figuring out what to display to a user at display-time is expensive.

Preindex (or "inbox") basically says, pick who sees a piece of content when the content is created, and add them to an index. The good news is that display is fast an easy (simple index walk). The bad news is that changes to follow lists are not automatically retroactive. Still, if you want scalability, this is pretty much your only choice.

Tumblr switched from scatter-gather to inbox for their feed ("Dashboard Index"): http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html

In an oldie-but-goodie Google I/O talk from 2009, Brett Slatkin talked about how to build an inbox-like million-user-fanout on GAE: https://www.youtube.com/watch?v=AgaL6NGpkB8

Side note: The Objectify google group is fairly quiet because most questions are now asked on stackoverflow (for better or worse). Also, you're looking at very old documentation for an obsolete version of Objectify. Current documentation is here: https://github.com/objectify/objectify/wiki