Export GCP Datastore and import to a different GCP Project

Question

I'm trying to copy data from DataStore in one GCP project to another - the process doesn't seem to give any errors, but also doesn't populate my target datastore.

In my source project I create a bucket and export my datastore in to it as follows :

gcloud datastore export gs://ctb_datatransfer

I then switch to my destination project and run

gcloud datastore import gs://ctb_datatransfer/2020-03-01T18:41:37_32872/2020-03-01T18:41:37_32872.overall_export_metadata

Neither of these steps seem to indicate any errors, but my destination datastore remains empty after the import has completed.

Any help is gratefully received as I don't really have much of a clue about what is happening - I believe that I have granted the correct permissions on the bucket ( and would expect to see some sort of permissions failure if I hadn't I guess )

The output from the export step looks like this

    done: true
metadata:
  "@type": type.googleapis.com/google.datastore.admin.v1.ExportEntitiesMetadata
  common:
    endTime: "2020-03-01T18:41:50.349159Z"
    operationType: EXPORT_ENTITIES
    startTime: "2020-03-01T18:41:37.188012Z"
    state: SUCCESSFUL
  entityFilter: {}
  outputUrlPrefix: gs://ctb_datatransfer/2020-03-01T18:41:37_32872
  progressBytes:
    workCompleted: "8848"
    workEstimated: "6480"
  progressEntities:
    workCompleted: "40"
    workEstimated: "40"
name: projects/chattyappsdata-staging/operations/ASAzMTEwMzAzMTMJGnRsdWFmZWQHEjJ3LXVlLXNib2otbmltZGEQCigS
response:
  "@type": type.googleapis.com/google.datastore.admin.v1.ExportEntitiesResponse

while the output from the import step looks like this

    done: true
metadata:
  "@type": type.googleapis.com/google.datastore.admin.v1.ImportEntitiesMetadata
  common:
    endTime: "2020-03-01T18:46:25.251008Z"
    operationType: IMPORT_ENTITIES
    startTime: "2020-03-01T18:45:44.029056Z"
    state: SUCCESSFUL
  entityFilter: {}
  inputUrl: gs://ctb_datatransfer/2020-03-01T18:41:37_32872/2020-03-01T18:41:37_32872.overall_export_metadata
  progressBytes:
    workCompleted: "8110"
    workEstimated: "8848"
  progressEntities:
    workCompleted: "40"
    workEstimated: "40"
name: projects/chattytoolbox/operations/AiAwNTEwNDAxMTMJGnRsdWFmZWQHEjJ3LXVlLXNib2otbmltZGEQCigS
response:
  "@type": type.googleapis.com/google.protobuf.Empty

How are you testing for an empty datastore? Are you looking in the cloud console for chattytoolbox? — Jim Morrison
@JimMorrison - yes - just refreshing the console page and it tells the the database is empty — DaveH
@JimMorrison - actually - and wierdly - I've just checked it now ( after not doing anything for 24 hours or so ) and the entities have now appeared in the destination datastore. — DaveH
@Ajordat - well, the data now exists in the destination datasource but it seemed to appear without me doing anything. I don;t know if this is just an artefact of the GCP datastore console, but it seemed like the immediately after I did the imprt the console told me the database was empty, and then when I checked again several hours later, the data had appeared. I don't understand why or how that happened. — DaveH

Ajordat Ajordat · Accepted Answer · 2020-03-06T12:58:13

As stated by OP in the comments, the whole process ended up working, it just did it hours later. This is a possible explanation:

Looking at the documentation about importing entities, I've found that the import is a long running operation (LRO) and thus it may take a substantial amount of time to complete. From the import output you shared, we can see that the operation did finish, but I believe it was missing some propagation, which got fixed by time.

Also, the import command (gcloud datastore import) it just starts the LRO. If you specify the argument --async the command will return immediately but if you don't, it will wait for a response. On this section, on the gcloud tab, it's stated that even if the import command gets cancelled the operation will continue, thus reinforcing the theory about the independence of the command, the import and the propagation.

Export GCP Datastore and import to a different GCP Project

1 Answers