Template file failed to load with Dataflow bulk delete api

Question

I'm using the REST API template provided from https://cloud.google.com/dataflow/docs/guides/templates/provided-utilities#api_2 to delete from datastore but I'm getting the following error

"error": {
  "code": 403,
  "message": "(55ae8f210de971e7): Template file failed to load: gs://dataflow-templates/dataflow-templates/latest/Datastore_to_Datastore_Delete. Permissions denied. Provided scope(s) are not authorized",
  "status": "PERMISSION_DENIED"
}

It says the scope isn't authorized but the docs say you only need one of the following: https://www.googleapis.com/auth/compute.readonly, https://www.googleapis.com/auth/compute, https://www.googleapis.com/auth/cloud-platform or https://www.googleapis.com/auth/userinfo.email.

For my oauth2 request I tried adding the devstorage scope as well: https://www.googleapis.com/auth/compute.readonly https://www.googleapis.com/auth/devstorage.read_only

I'm calling POST https://dataflow.googleapis.com/v1b3/projects/{projectid}/templates:launch?gcsPath=gs://dataflow-templates/latest/Datastore_to_Datastore_Delete
With

{
    jobName: 'PrunePrintLogs',
    environment: { zone: 'europe-west2' },
    parameters: {
      datastoreReadGqlQuery: 'select * from `00000000test`',
      datastoreReadProjectId: '{projectid}',
      datastoreDeleteProjectId: '{projectid}'
    }
  }

My project id is redacted.

I added the Owner role to the account getting the oauth2 token just temporarily for testing so wouldn't all scopes be authorized?

Terry King Terry King · Accepted Answer · 2021-07-13T13:57:38

Got it working. Firstly I needed to remove the 'environment' parameter. Secondly I set scope to 'https://www.googleapis.com/auth/cloud-platform'. It also seems the role needs to be Editor, I wasn't able to find a more restrictive role to get it working.

Curiously when I sent the request from postman is shows up in Dataflow jobs as sdk 2.27.0 but I'm sure I'm sending exactly the same from my node app using axios and it shows up as sdk 2.20.0 and warns that it will eventually be no longer supported.

Template file failed to load with Dataflow bulk delete api

1 Answers