Entity Group - deciding on how to group

2
votes

I've read throughout the Internet that the Datastore has a limit of 1 write per second for an Entity Group. Most of what I read indicate a "write to an entity", which I would understand as an update. Does the 1 write per second also apply to adding entities into the group?

A simple case would be a Thread where multiple posts can be added by different users. The way I see it, it's logical to have the Thread be the ancestor of the Posts. Thus, forming a wide entity group. If the answer to my question above is yes, a "trending" thread would be devastated by the write limit.

That said, would it make sense to get rid of the ancestry altogether or should I switch to the user as the ancestor? What I'd like to avoid is having the user be confused when they don't see the post due to eventual consistency.

1
google-app-enginegoogle-cloud-datastore
Have you read this: cloud.google.com/datastore/docs/articles/… - voscausa
Great read. It explains the concepts to consider very well. However, as with other articles, the example used regarding write limits was for updates of entities. - Rey Pader
What's "logical" isn't always the right answer with Datastore. You need to think about the queries and updates that your application needs. If you need strong consistency, then, yes, having Thread as a Post's ancestor might make sense. But do you need strong consistency? If not, don't use an ancestor and you'll not suffer the same write contention. - tx802
worth pointing is that you can get strong consistency without ancestors, by doing "get by key" queries - Patrice

1 Answers

1
votes

A quick clarification to start with

1 write per second doesn't mean 1 entity per second. You can batch writes together, up to a maximum of 500 entities (transactions also have a 10 MiB limit). So if you can patch posts, you can improve your write rate.

Note: you can technically go higher than 1 per second, although your risk of contention errors increases the longer you exceed that limit as well as the eventual consistency of the system.

You can read more on the limits here.

Client-side sharding

If you need to use ancestor queries for strong consistency AND 1 write per second is not enough, you could implement client-side sharding. This essentially means that you write the posts to a up to N different entity-groups using a known key scheme, For example:

  • Primary parent: "AncestorA"
  • Optional shard 1: "AncestorA-1"
  • Optional shard N: "AncestorA-(N-1)"

To query for your posts, issue N ancestor queries. Naturally, you'll need to merge these results on the client-side to display it in the correct order.

This will allow you to do N writes per second.