Storing user data in Google Cloud Datastore as an Entity Group

Question

I am trying to find the best way to store a user in google datastore (using nodejs). I have the userId and a userData object:

userId: "some-random-string"

userData: {
    info: {
        name: "name",
        email: "[email protected]"
    },
    profile: {
        username: "user",
        status: "using"
    },
    stats: {
        stuffDone: 12,
        stuffNotDone: 20
    },
    inventory: {
        money: 100,
        items: ["first", "second", "third"]
    }
}

I know I can store all this as a single entity but would it be worth it to split it up into an entity group if I will be updating all the nested objects separately (info, profile, stats, inventory).

So I would have the root entity (that probably wouldn't exist):

datastore.key(["Users", userId])

then I would create 4 children to store the userData:

datastore.key(["Users", userId, "UserData", "Info"); --> userData.info
datastore.key(["Users", userId, "UserData", "Profile"); --> userData.profile
datastore.key(["Users", userId, "UserData", "Stats"); --> userData.stats
datastore.key(["Users", userId, "UserData", "Inventory"); --> userData.inventory

Only the user would be updating the data so contention should not be an issue. Once the user is created I wouldn't need to update more than one child at a time.

So say the stats is updated every minute, I can just update it directly with the key:

datastore.key(["Users", userId, "UserData", "Stats");

Would this be the best practice to split it up instead of rewriting the whole user object to a single entity and have to rewrite all the indexes?

With the entity group I can still query all the user data at once with:

query = datastore.createQuery().hasAncestor(datastore.key(["Users", userId]));

Then I would just need to process it to get it back into the userData object above. I would only need to do this once when the user logs in, all other times I would need to get user data it would only be a single child and I could just get the child by key.

If I shouldn't be using an entity group like this then I could do the same thing by storing each part of the user in separate entities like:

datastore.key(["UsersInfo", userId); --> userData.info
datastore.key(["UsersProfile", userId); --> userData.profile
datastore.key(["UsersStats", userId); --> userData.stats
datastore.key(["UsersInventory", userId); --> userData.inventory

Then I could still update them individually but I think would be more taxing to get all the data since I would need to do 4 queries instead of an ancestor query.

Would these entity groups or multiple entities be necessary if I am only updating the userData.stats and userData.profile around once per minute, or should I just be using a single entity. The stats and profile objects will get bigger than only a couple properties.

Dan Cornilescu Dan Cornilescu · Accepted Answer · 2018-06-13T03:23:57

Splitting an entity into multiple related ones based on the property update patterns may be a good idea, especially for large entities - to avoid unnecessarily re-writing the entire entity when just a portion of it changes (with the associated increase in the duration of the update of the entity itself and of all its related indexes. See related re-using an entity's ID for other entities of different kinds - sane idea?

Placing all the entity "pieces" into the same entity group - which, as you observed, allows you to make ancestor queries to retrieve the user data, is OK as long as you respect the max 1 write/second per entire entity group. Just double-check if you can safely assume that the query results come in a specific order, otherwise you may need to take steps to ensure each result goes to the right object.

Using separate, non-ancestor related entity "pieces" allows a higher overall write rate than the split with ancestor: max 1 write/second per each entity. And technically you don't need to do 4 queries to retrieve the entities, but 4 key lookup operations, see Retrieving an entity.

But splitting the entity can also increase your datastore costs: the single read/write operation for a single entity will be multiplied by the number of "pieces" in which the entity was split - in your case 4 if you read/write all of them together.

So you'll have to balance these pros and cons in the context of your application.

Storing user data in Google Cloud Datastore as an Entity Group

1 Answers