Datastore entity key collision

Question

In development I have an empty datastore. Online I have a datastore with millions of entities.

In the development datastore (locally) I put one new entity (generating a new key). Then I export the entity and put it in the online datastore (with the key generated locally). What is the risk that the key generated locally to have already been assigned to an entity in the online datastore?

Or would it be simpler to avoid collision by creating the keys locally like this:

for (int i = 0; i < data.size(); i++) {
    Key k = KeyFactory.createKey(kind, new Date() + i);
    // continue to creating and inserting entities...
}

Thanks.

Answer 1

From https://developers.google.com/appengine/docs/java/datastore/entities:

Instead of using key name strings or generating numeric IDs automatically, advanced applications may sometimes wish to assign their own numeric IDs manually to the entities they create. Be aware, however, that there is nothing to prevent the Datastore from assigning one of your manual numeric IDs to another entity. The only way to avoid such conflicts is to have your application obtain a block of IDs with the methods DatastoreService.allocateIds() or AsyncDatastoreService.allocateIds(). The Datastore's automatic ID generator will keep track of IDs that have been allocated with these methods and will avoid reusing them for another entity, so you can safely use such IDs without conflict.

You must either generate all of the numeric keys manually (and in such a way that they will never collide) or use allocateIds(). There is specifically no guarantee that anything you generate manually will not collide with an existing key, unless you use that function. The generated key ids are not like an auto increment field in a relational db which increment by one each time.

Answer 2

It would depend on the kind of IDs you have on the entity. If they are integers, they are small and you can easily have conflicts.

I would suggest just creating a slightly different key format from what Google uses by default. Your function above looks like it should work as long as the KeyFactory uses the full accuracy of the clock (i.e. does not stop at seconds).

Answer 3

When the datastore generates keys the ids are integers. If you generate keys using Strings as Ids then they will not override the existing entities.

a.key.id() == 1
b.key.id() == '1'
a.key.id() != b.key.id()

Answer 4

You should provide a bit more information about your use case for anyone to suggest a proper solution. You could for instance create keys on the dev database like this:

  KeyFactory.createKey(entity, "dev_" + UUID.randomUUID())

Another option would be to create a small utility or app that will retrieve the Entity from the local database and then POST it to the development database. The online server getting the POST would create a new Entity and assign all properties from the client except the key, thereby letting GAE generate a new key automatically. The new key could then be sent back to the client, as a response to the POST operation.

Again, I don't know if you later want to update the local database an resync with the online, but if so, the first approach with defining your own key would work as the next sync will update the entity online.