Deleting or updating Datastore properties using Google Cloud Dataflow

Question

If you want to add a property to an existing entity, you just "clone it" and add it.

  Entity oldEntity = c.element();
  Entity.Builder entityBuilder = Entity.newBuilder(oldEntity);
  entityBuilder.addProperty(
          DatastoreHelper.makeProperty("newProperty",
                  DatastoreHelper.makeValue("Value")
          )
  );
  c.output(entityBuilder.build());

If you want to update a property, adding it again hoping to overwrite the old value won't work. It won't save to the datastore because a property's name must be unique, and now you'd have two with the same name.

Error writing to the Datastore (400): Entity has duplicate property name

If you want to remove a property, you need to know the index from the Property List, and for that you'd need to list all the properties, check if the property you want to update exists, keep track of the index number, and then remove it.

Is there a builtin helper for this procedure or a shortcut I'm missing?

Caio Iglesias Caio Iglesias · Accepted Answer · 2016-06-16T10:08:33

Currently Google Cloud Dataflow's Java SDK uses Datastore API v1beta2 and there's no way to directly add a property to an entity, not even using DatastoreHelper.getPropertyMap and adding properties to the resulting Map<String, Value> because that method returns an UnmodifiableMap.

When they switch to v1beta3 the properties will be exposed as just a map, according to a team member.

So this is how I managed it on v1beta2:

Entity oldEntity = c.element();

// We need to get the property map, but the one from DatastoreHelper is an unmodifiableMap
Map<String, Value> oldEntity_map = DatastoreHelper.getPropertyMap(oldEntity);
Map<String, Value> newEntity_map = new HashMap<String, Value>();
newEntity_map.putAll(oldEntity_map);

// Adding or updating a property
newEntity_map.put("newProperty", DatastoreHelper.makeValue("Value").build());
// Deleting a property
newEntity_map.remove("delete-this");

Entity.Builder updatedEntity = Entity.newBuilder(oldEntity);
updatedEntity.clear();
updatedEntity.setKey(oldEntity.getKey());

for (Map.Entry<String, Value> property : newEntity_map.entrySet())
{
    updatedEntity.addProperty(
       DatastoreHelper.makeProperty(property.getKey(), property.getValue()));
}

c.output(updatedEntity.build());

Deleting or updating Datastore properties using Google Cloud Dataflow

1 Answers