GCP Cloud Scheduler throws ERROR for a HTTP targettype

4
votes

I have created a GCP Cloud Scheduler job to run every 15 minutes. It is supposed to call an API from my Node js application. In the console the job definition looks like this:

Description: A job
Frequency: */15 * * * *
Timezone: Central Standard Time
Target: HTTP
URL: https://<company url>/api/email-reminder/
HTTP method: GET
Auth header: Add OIDC token
Service account: xxxxxxxxxxx-compute@developr.gserviceaccount.com

When it runs it returns the following in the logs:


httpRequest: {
 }
 insertId: "15wxxxxxxge1lv"  

jsonPayload: {
  @type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"   
  jobName: "projects/<project name>/locations/us-central1/jobs/xxxxxxxxx-scheduler-emailreminders-1"   
  status: "UNKNOWN"   
  targetType: "HTTP"   
  url: "https://<company url>/api/email-reminder/"   
 }
 logName: "projects/<project name>/logs/cloudscheduler.googleapis.com%2Fexecutions"  
 receiveTimestamp: "2019-11-14T04:45:50.280446452Z"  

resource: {

labels: {…}   
  type: "cloud_scheduler_job"   
 }
 severity: "ERROR"  
 timestamp: "2019-11-14T04:45:50.280446452Z"  
}

How do I find out more information about the error?

3
google-cloud-platformgoogle-cloud-scheduler
What does StackDriver log say? It might give you some hint – Prashant
The second half of my post is the StackDriver log entry for the job run. – Bill Soranno
What happens what you call the endpoint manually with curl? Configure the service account in the CLI gcloud. Then get an Idenity Token gcloud auth print-identity-token. Then execute curl -H 'Authorization: Bearer ID_TOKEN' https://<company url>/api/email-reminder/ – John Hanley
I am assuming that the mistake in the service account is a typo in your question and not in the Cloud Scheduler task definition: @developr – John Hanley
Can you provide the list of role of the used service account ? – guillaume blaquiere

3 Answers

8
votes

The default value for timeout scheduler jobs process is 180s, you can change vía gcloud command

gcloud scheduler jobs update http my-super-job --attempt-deadline 540s

As well you can see the complete info of your jobs with this commands...

gcloud scheduler jobs list
gcloud scheduler jobs describe my-super-job
0
votes

I've seen similar problems recently using Cloud Scheduler for HTTPS targets. Every so often, the scheduler will fail, and all I get is a log message like yours.

Viewing in the Logs Viewer, the key parts of the log are, in the log header:

"status":"RESOURCE_EXHAUSTED",
"@type":"type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"

and in the log data:

httpRequest: {
  status: 429   
}
jsonPayload: {
  @type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"   
  jobName: "projects/joburlhere"   
  status: "RESOURCE_EXHAUSTED"   
  targetType: "HTTP"   
  url: "https://urlgoeshere"   
}
severity: "ERROR"

"Resource exhausted" is the description of the 429 error code.

There's a description of this code here:

https://cloud.google.com/apis/design/errors

429 RESOURCE_EXHAUSTED Either out of resource quota or reaching rate limiting. The client should look for google.rpc.QuotaFailure error detail for more information.

Given that I'm running this Job once an hour, and the receiver is a very modest Cloud Function, I don't I'm doing anything to cause resource exhaustion. So I think it's a recurring transient problem with Google's cloud infrastructure. I'm guessing that the cloud functions were unavailable for that particular request, and because I set up the Cloud Function with default settings, the Scheduler did not retry.

Additionally, it is possible to configure a Scheduler Job to retry in case of failure. This functionality is not shown in the web console, but you can control it using the gcloud command.

The default setting is not to retry.

Look at the --max-retry-attempts flag.

https://cloud.google.com/sdk/gcloud/reference/scheduler/jobs/update/http

There are similar controls for pubsub Jobs

https://cloud.google.com/sdk/gcloud/reference/scheduler/jobs/update/pubsub

0
votes

It's because the http endpoint you specified doesn't return the response within the default attempt deadline. Refer the link