Deleted entities keep returning using datastore admin

2
votes

I am trying to copy the datastore from one app to another (live to dev server). I first need to delete all the entities on the dev server (the actual dev app I'm using, not to be confused with the local dev server). I am doing so in the datastore admin console, but after the process completes, I still have _GAE_MR_MapreduceState and _GAE_MR_ShardState entities remaining. These should be temporary during the delete process as far as I understand. Besides this, occasionally when I go to the datastore admin console (about half the time) I also see a number of my deleted entities also showing up again. I have deleted them again a couple times.

I don't see them in the datastore viewer, but I don't know if this is always the case. I've just yet to see them show up in this view again. In datastore viewer there are only entities of type _AE_Backup_Information, _AE_Backup_Information_Kind_Files, _AE_Backup_Information_Kind_Type_Info and _AE_DatastoreAdmin_Operation. I don't see _GAE_MR_MapreduceState and _GAE_MR_ShardState entities in this view.

UPDATE: I just was able to catch one of my deleted entities in the datastore viewer. Although there was only one of the deleted entity types and only one of the entities visible here (before I was seeing 3 or 4 entity types and thousands of entities). Right after I caught it in the datastore viewer I went to the datastore admin console and found that the same entity type was visible here, although two of them in this case. I also didn't find _GAE_MR_MapreduceState and _GAE_MR_ShardState nor _AE_Backup_Information, _AE_Backup_Information_Kind_Files, _AE_Backup_Information_Kind_Type_Info and _AE_DatastoreAdmin_Operation entities in the datastore admin this time. END UPDATE

I have not tried deleting the _GAE_MR_MapreduceState and _GAE_MR_ShardState entities in the datastore admin console. This seems kind of unsafe as these entities are created during a delete operation, I don't know what it would do. I would attempt deleting them in the datstore viewer, but they aren't showing up there.

Those times that my deleted entities show up in the datastore admin console _AE_Backup_Information, _AE_Backup_Information_Kind_Files, _AE_Backup_Information_Kind_Type_Info and _AE_DatastoreAdmin_Operation entities are also visible there as well. So this is strange too.

There are no tasks on the task queue and no jobs are running.

Is this some sort of synchronization issue? Is a cache not being updated somewhere? Is there some way I can reset something to get these entities cleared out?

UPDATE 2: I tried flushing Memcache and this did not make a difference. The entities continue to show up in the datastore admin console, pretty much 100% of the time. I have not seen any of the deleted entities show up in the datastore viewer however since I deleted the one I caught showing up before.

UPDATE 3: I just attempted to perform the same type of datastore delete operation in another app and I am getting the same corruption as a result. I first created a backup, then selected and deleted all entities, both operations from the datastore admin console.

The delete operation jobs completed. They show up logged under completed operations. No more jobs are left in the task queue. But all of the entities are visible again in the datastore admin console! I deleted 10 entity types, most with a couple dozen entities, one with several thousand. Every one of the entity types is listed again with the same number of entities in the datastore admin console.

In the datastore viewer, two of the entity types still remain, with a smaller (but still significant) number of entities than before the delete operation. One with 2,172 from 6,855 before the delete. The other with 17 from 44 before the delete.

Again I flushed the Memcache. It made no difference.

I see that the datastore admin is experimental, but I wouldn't expect it to completely break my datastore by performing a simple delete operation with no way to reset the datastore afterwards. I now have two apps, both with billing enabled costing my client money, that are unusable because their datastores are corrupted. What are my options here? Is it possible I can get a response from someone at Google? This is the recommended forum for Google App Engine support.

1
google-app-enginegoogle-cloud-datastoredatastore-admin
why don't you just create a new dev server data store ? I know that doesn't solve the problem of entities turning up. – Tim Hoffman
Do you mean a new app? The datastore is associated with the app. I'd like to maintain my app version history and other history and settings associated with the current app as well as the name and URL, which would not be possible if I were to create a new app. Other than that, I don't know of a way to create a new datastore for an app. – Jeff Lockhart
You might want to set up a way to generate a datastore with representative data. Some combination of programmatically creating entities and copying over select entities from prod. You shouldn't have to deal with _GAE and _AE (private) entities. – Joel Burget
I don't understand your solution. I haven't gotten to copying the production datastore yet as I'm still trying to purge the old datstore entities from the dev app. They continuously keep showing up in the datastore admin console after being deleted. – Jeff Lockhart
@jeffdgr8 - no I meant create a new dev data store, if you start the devserver with the --clear_datastore option it will be empty. This wil have no impact on your code etc.. – Tim Hoffman

1 Answers

0
votes

It looks like you have old style exports set up in your dev app. You should probably check your cron.yaml and disable those exports. Ideally, you would switch to the newer (and cheaper) exports using the Managed Import/Export service. For bulk deletes the current recommendation is to use Cloud Dataflow with the bulk delete template. Both of the Managed Import/Export & Cloud Dataflow based delete will not use the Cloud Datastore database in your project, where both the old style exports and deletes will use your Cloud Datastore database.