How to create API methods in Google App Engine that have several decedents/ancestors

Question

I am having trouble understanding how to structure an ancestor tree with several decedents. Suppose I have a model like this (every Entity has a Long id):

User
    -Post
        -Comment

Where the Comment is the grandchild of the User.

What is really annoying is to insert a Comment I need to generate the Key of the Post. And to generate the Key of the Post I also need to know the ID of the User:

Key<Post> postKey = Key.create(Key.create(User.class, userId), Post.class, postId);

This is a problem for me because when trying to insert a Comment into the datastore I need to also pass in the userId and postId just to generate the key of the Post.

Similarly, it is annoying to try and get a one Post because I need to pass in both the userId and postId to generate the Key.

I am looking for a better way to structure my model and API methods without having to pass in all those ancestor IDs to my methods. I was considering storing the websafeKey in every Post and Comment entity as a property like this:

String websafeKey = Key.create(Key.create(User.class, userId), Post.class, postId).getString();
Key<Post> key = Key.create(websafeKey);

Then I could the key to every Post and Comment (and other children of these Entities)) right there in the Entity. Then presumably I wouldn't have to pass in all those ancestor IDs in to my API methods all the time.

Not sure if that is a good idea though.

Answer 1

Based on the info provided you have two options:

• Pass full keys around and design your API around that
• decouple your entities ancestor relationships, so that no entity has a parent

Note: after writing this all out, I realized the end API looks the same really either way:

GET /user/{key} - get user info
POST /user/{key}/post/ - create post 
GET /post/{key} - get post
POST /post/{key}/comment/ - create comment
GET /comment/{key} 

Full keys

In this case, {key} is the websafe key.

Advantages:

• you can maintain transactional and consistency control
• allows you to change parent entity relationship later without migrating old data and breaking old links (in my experience this is a massive win)
• Allows you to mix kinds (e.g. Have a Post kind and a Post2 kind, or an ImagePost kind - useful for polymorphism or for breaking migrations)

Decouple entities as ancestors

In this case {key} is the id, and there is no entity hierarchy

Dis/Advantages:

• Simplicity
• You need to infer the kind based on the URL
• listing posts for a user or comments for a post would always be eventual
• no migration path if you need to introduce transaction groups

• internal coupling for consistency and transactionality not exposed through API
• aligns with restful API as resource concept
• supports either model

In my experience you should absolutely do the first option, you will probably get your data model wrong a few times, and being able to change/migrate it is an absolute win.

Answer 2

First of all, I agree with Konqi when he says that you should carefully design your ancestor/keys model to get the right level of consistency that you want, taking in consideration the trade-offs of write/sec throughput.

Assuming that you still want the user -> post -> comment design, to deal with datastore ancestor key references like this, we came to a solution that is almost similar to your idea. We create string representations of the keys. For your specific case, it would be:

/users/{userId}/posts/{postId}/comments/{commentId}

We also have a convenience type to encapsulate that representation, called IdRef<T>.

Everytime an Entity is retrieved from the datastore store we construct this representation and for every API request we translate it from the given string.

Using that approach, we can easily expose APIs like that:

GET /users/{userId}/posts/{postId}/comments/{commentId}
POST /users/{userId}/posts/{postId}/comments/{commentId}
DELETE /users/{userId}/posts/{postId}/comments/{commentId}

I maintain an opensource project that, among other features, solves the problem you are mentioning. It is a Java DSL designed to expose RESTful APIs from your appengine datastore models.

If you are interested, here is a gist that exemplifies your use case.

Answer 3

I ran into the same exact problem, and have currently implemented something similar to @feroult, though I am still looking for a better method. Let me explain some of the logic behind this decision.

Let's assume we read a list of some resource that is at least a few levels down in the tree. The "comment" example above could be one example. In this case, if we were to use datastore keys on the client side, then yes, our API would be very RESTful in that the key itself would encompass the path to the root entity. The problem however is that this method is inefficient since the ancestor path is repeated in every key.

The alternative is to create a tree like structure on the client that resembles the tree like data from the datastore. If we want all the comments made by a user, then we only need to have the user id reside in memory once. There is no reason to have it represented in a key for every comment. Which then leads us to the ugly mess of sending the full ancestor path /users/{userId}/posts/{postId}/comments/{commentId} in the RESTful api.

At the end of the day, no matter how the actually path looks, the same information must be passed to the datastore. This is critical. Google Datastore cannot find a comment without the ancestor path.

This means that the problem can be reduced to one of semantics. Perhaps the compromise is to create an equivalent client side key factory that can generate a key from the ancestor ids and then send it to the server. In this case, repetition would be avoided AND the RESTful API would still be clean.

Would be interested in hearing other thoughts.