options for restoring appengine datastore data?

Question

A user of our application has accidently deleted data. They'd like this to be restored. We have no special logic or datastore entities that can do this.

However, we do daily backups of our entire datastore to blobstore using the datastore admin.

What are our options for selectively restoring part of this backup back into the datastore?

We'd preferably like to not have a service interruption for other users. One final restriction is that we can not change our production app id (i.e. copy data over to a new app and then restore the backup to our old app - this is because our clients reference our appid directly).

Thoughts?

UPDATE

I was thinking of running a mapreduce over all the blobs in our app and finding the ones that are to do with our backup. Parsing these backups and restoring the entities as needed. The only issue is, what format are the blobs stored in? How can I parse them?

Answer 1

Since 1.6.5 the Datastore Admin now allows you to restore individual Kinds from an existing backup.

About the backup format: according to the datastore admin source code you can use RecordsReader to read backup file stored in leveldb log format in a MapperPipeline

Answer 2

The restore functionality in its current form is not very useful for my application. There should be an option to restore only few entities or namespaces into current app-id or another app-id. Please star this issue http://code.google.com/p/googleappengine/issues/detail?id=7311

Answer 3

May be custom backup reader help you

final BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
final BlobKey blobKey = blobstoreService.createGsBlobKey("/gs/" + bucket + "/" + pathToOutputFile);

final RecordReadChannel rrc = BlobserviceHelper.openRecordReadChannel(blobKey, blobstoreService);

ByteBuffer bf;
while ((bf = rrc.readRecord()) != null) {
  final OnestoreEntity.EntityProto proto = new OnestoreEntity.EntityProto();
  proto.mergeFrom(bf.array());
  final Entity entity = EntityTranslator.createFromPb(proto);
  entity.removeProperty(""); // Remove empty property
  //Now you can save entity to datastore or read keys and properties
}