Schedule Dataflow batch jobs with Cloud Scheduler - Permission Denied

0
votes

as the title said, i'm trying to run dataflow with scheduler based on tutorial on this link https://cloud.google.com/community/tutorials/schedule-dataflow-jobs-with-cloud-scheduler

i think i already followed the exact step on the tutorial (just different region config) and the scheduler is created but when i'm trying to run the scheduler, i'm getting this error

{ "status": "PERMISSION_DENIED", "jobName": "projects/my-project-id/locations/asia-south1/jobs/scheduler-demo", "url": "https://dataflow.googleapis.com/somerandomidthatidontknow/projects/my-project-id/locations/asia-south1/templates:launch?gcsPath=gs://my-testing-bucket/templates/dataflow-demo-template", "@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished", "targetType": "HTTP" }

is it because of the region? because the tutorial said

Cloud Scheduler jobs must be created in the same region as App engine.

my app-engine is on asia-south1

my scheduler is on IST timezone

and my bucket is on US multiple regions in united states

service account: [email protected]

  1. Dataflow Admin (default role created when running terraform from tutorial)
  2. Dataflow Worker(trying to fix this issue by adding this, but not working)
  3. Owner(trying to fix this issue by adding this, but not working)
2
google-cloud-platformgoogle-cloud-storagegoogle-cloud-scheduler

2 Answers

0
votes

It seems that your service account used for Cloud scheduler doesn't have right permission.

http_target {
    http_method = "POST"
    uri = "https://dataflow.googleapis.com/v1b3/projects/${var.project_id}/locations/${var.region}/templates:launch?gcsPath=gs://${var.bucket}/templates/dataflow-demo-template"
    oauth_token {
      service_account_email = google_service_account.cloud-scheduler-demo.email
    }

According to the link you said, You should specify service account email your own.

service_account-email = google_service_account.cloud-scheduler-demo.email

You can give your service account a role to run dataflow job successfully.

0
votes

Here's the step to fix this permission denied issue

  1. Find the project number for your project on the Google Cloud Project Settings Page
  2. Copy down the number
  3. Open the IAM Admin Console Page
  4. Click Add. The Add members screen opens
  5. In the New members dialog box, add an email address of the format: service-[project-number]@gcp-sa-cloudscheduler.iam.gserviceaccount.com
  6. Replacing [project-number] with your project number from above
  7. From the Select a role drop-down, choose Service Management -> Cloud Scheduler Service Agent -> save

i found the solution by following step on this link https://cloud.google.com/scheduler/docs/http-target-auth