Google Cloud Tasks enforcing rate limit on forwarding to Cloud Functions?

0
votes

Cloud Tasks is saying:

App Engine is enforcing a processing rate lower than the maximum rate for this queue either because your application is returning HTTP 503 codes or because currently there is no instance available to execute a request.

However, I am forwarding the tasks to a cloud function using an HTTP POST request, similar to the one outlined in this tutorial. I don't see any 503s in my logs for the cloud function that it forwards to.

My queue.yaml is:

queue:
- name: task-queue-1
  rate: 3/s
  bucket_size: 500
  max_concurrent_requests: 100
  retry_parameters:
    task_retry_limit: 1
    min_backoff_seconds: 120
    task_age_limit: 7d

enter image description here

1
google-cloud-platformgoogle-cloud-functionsgoogle-cloud-tasks
What the problem? The enforce rate is too low, right? If so, how many task do you have stacked in your queue? - guillaume blaquiere
@guillaumeblaquiere I had about 64k tasks in the queue. - Matthew Mage
Do you have logs of the called service? Everything is OK? - guillaume blaquiere
Yes. I don't see any 503s in my logs for the cloud function that it forwards to. It is successfully processing the tasks, just really slowly. I am now down to 28k tasks at the moment from 64k yesterday. Still at a reduced rate. - Matthew Mage
Any other code except 200 OK ? Some very long request? - guillaume blaquiere

1 Answers

0
votes

The problem seems to come from any exception, even though only 503 is listed. If the cloud function responds with any error the task queue slows down the rate, and you have no control over that.

My solution was to swallow any errors to prevent that propagating up to Google's automatic check.