Cannot set dynamic template when posting to Cloud Dataflow template REST API

Question

I am trying to schedule a dataflow job with Cloud Scheduler by posting to REST API and authenticate by OAUTH. I have created a template and it works when I manually use Dataflow 'Create job from template'. However, when being used as a HTTP endpoint (https://dataflow.googleapis.com/v1b3/projects/${my.proj}/locations/europe-west1/templates:launch?gcsPath=gs://${my.proj}/templates/${template.name}), it returns the following error:

{
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unknown name \"jobName\": Cannot bind query parameter. Field 'jobName' could not be found in request message.\nInvalid JSON payload received. Unknown name \"environment\": Cannot bind query parameter. Field 'environment' could not be found in request message.\nInvalid JSON payload received. Unknown name \"parameters\": Cannot bind query parameter. Field 'parameters' could not be found in request message.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "description": "Invalid JSON payload received. Unknown name \"jobName\": Cannot bind query parameter. Field 'jobName' could not be found in request message."
          },
          {
            "description": "Invalid JSON payload received. Unknown name \"environment\": Cannot bind query parameter. Field 'environment' could not be found in request message."
          },
          {
            "description": "Invalid JSON payload received. Unknown name \"parameters\": Cannot bind query parameter. Field 'parameters' could not be found in request message."
          }
        ]
      }
    ]
  }
}

Following is my POST body:

{
  "jobName": "test",
  "parameters": {
    "region": "europe-west1"
  },
  "environment": {
    "tempLocation": "gs://${my.proj}/temp",
    "zone": "europe-west1"
  }
}

I appreciate any help, thank you in advance!!

How d you call the API? with curl? with your code? Can you share this part? — guillaume blaquiere

Peter Kim Peter Kim · Accepted Answer · 2020-08-24T21:29:55

I verified that below works:

TEMPLATE_LOCATION="gs://${my.proj}/templates/${template.name}"
API_ROOT_URL="https://dataflow.googleapis.com/"
TEMPLATES_LAUNCH_API="${API_ROOT_URL}/v1b3/projects/${my.proj}/locations/europe-west1/templates:launch"

time curl -X POST -H "Content-Type: application/json" \
 -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 "${TEMPLATES_LAUNCH_API}"`
 `"?gcsPath=${TEMPLATE_LOCATION}"`
 ` -d ' 
 {
    "jobName": "test",
    "parameters": {},
    "environment": {
        "tempLocation": "gs://${my.proj}/temp",
        "workerZone": "europe-west1-d",
    }
}'

Note that you don't need to specify region in parameters, and it's workerZone instead of zone.

Cannot set dynamic template when posting to Cloud Dataflow template REST API

1 Answers