Google AppEngine Entity group sizes and eventual consistency

Question

I have a shopping cart application that contains users, accounts, orders, order lines etc entities.

Many of my entities belong to the account entity e.g. an account has many orders, an order has many order lines and an order line has as many-to-many shipments relationship. The account also has many users who can view the account's child entities.

The documentation recommends keeping entity groups no larger than a single user's worth of data: https://developers.google.com/appengine/docs/python/datastore/entities

I'm concerned that an entity group for an account could grow to a size which becomes unscalable. An account could grow to 100,000 orders with hundreds of thousands of child entities.

I have two questions:

1.) If I don't use ancestors do I simply have to accept that if one user edits an entity it may not be updated for a few seconds?

2.) If I do use ancestors, what will happen if an account has many users all creating/editing/deleting within the same entity group throughout the day? Will some transactions get blocked?

Answer 1

Seems like the entity group here should be the order plus its items, not the account.

There is a limit of one update per second per entity group, so if you defined the group at the account level and you do have multiple users updating a single user constantly, some will fail. That's much less likely to happen if you define it as the order.

I would however strongly recommend that you do use an entity group for the order itself. You definitely wouldn't want there to be possible discrepancies in what the user sees - or worse, pays for - within a single order.

The only issue with doing things this way is that a new order might not be shown in the list of orders for an account immediately after it is created. To get round that you might use memcache, or potentially have a separate entity storing the list of orders for the user with a manually-specified key that you can get directly rather than querying - eg using the customer's account number as the key name. Since gets are guaranteed to be consistent, this would always be up to date.

Answer 2

1. Yes, but eventual consistency affects only queries. If you edit the data and then GET it, the changes will be visible.

2. No, there is no mention of size limitations or transaction blocking on entity groups mentioned in the docs. The limitation that applies is write throughput per entity group: docs state a limit of 1 write/second per entity group (in practice I see about 5 writes/sec). But this should not be an issue if you design your entity group to be per-user.