Google App engine datastore real-world application example

Question

The Problem

After days of reading the documentation of the app engine and the datastore I couldnt seem to understand how to build a real-world scalable application.

nor could I find any good examples for best practice. sure I can make my application work but I cant know if it is any good for scaling to more than the load my development environment can supply.

it clearly states on the app-engine tutorial of using the datastore that: "However, the rate at which you can write to the same entity group is limited to 1 write to the entity group per second. When you design a real application you'll need to keep this fact in mind."

Assumptions

I assume as for what I understand (please correct me if I am wrong) that given the datastore root is / then /Users/* (Keys) is an entity group and /Products/Beauty/* is an entity group which are effected by the previous limit of 1 write per second. (Is /Products/Beauty/* is limited by /Products/* aswell?)

Did I miss understand this?

Example

Lets assume we want to build a real-world chat application or a shopping application. these apps will have to store data for many operation that the user will do on the application. like sending a message or commenting on a Product or Deleting/Adding/Editing a Product.

given these operation how will the datastore should look like? Surely we cannot store the Messages as /Messages/{Message,To,From,Timestamp} because many users will send a message that will have to be written to this key. an alternative is to limit a user per 1 message per second and store it in his profile /Users/{USER}/{Message,To,From,Timestamp} but then only a single user can write to this entity group. so we can put them on the root as: /{USER}/{Message,To,From,Timestamp} but is it considered a good practice? is it fast enough to query the datastore to get all messages between users? (efficient? possible?)

another alternative is to store everything in the root of the datastore as "Messages" and "Users" and "Products" and "Categories" (for example a product will have a category reference in it).

Summary

1. Where can I find code-samples or design manual that are good for even millions of users?
2. Am I wrong with something here?
3. Any suggestions that are related to the topic?

I found these code samples but I have no tools or idea how to know if these are any good practices for scale?

Thanks!

Answer 1

You are thinking about it wrong. You Key paths would not be /Users/* (Keys)

But a set of ('User',somekey) keys, the user in this case is the Kind (representing an entity type). In this case each record is the root of it's own entity group.

You would only keep deep key chains (ancestors) where you have very tight requirements on owned relationships.

I have an application with in excess of 2000 users. And course material. Each user is it's own entity group owning it;s own results, with key references to the course material Each course owned by a training entity is in the same entity group (no requirement for hi write rate), and we see no reason why it won't scale to millions of users. Performance has scaled well when we went from 100 -> 2000 users.

Answer 2

Typically, an entity group is used only when you need transactions and/or strong consistency for entities in a single group. And the datastore does not gave a "root".

Let's look at the photo app as an example. When you add a new user, each user entity becomes its own "entity group", because it does not have (and does not need) a parent entity. This means that you can add as many users per second as you want.

When you add a new photo album, you can either make it a child entity of a user entity or you can add it without specifying a parent entity and simply reference a user in one of its properties. And if you decide to use a parent-child relationship between users and photo albums, you will be limited to adding one album per user per second. In a real life situation a user cannot add albums that fast.

App Engine is built with scalability as its key feature. Start building your app, or study one of the tutorials/sample apps, and many of these data modeling concepts will become clear.