How to auto-start an App-Engine backend when a pull-queue has tasks?

Question

It looks like I can create a push-queue that will start backends to process tasks and that I can limit the number of workers to 1. However, is there a way to do this with a pull-queue?

Can App-Engine auto-start a named backend when a pull-queue has tasks and then let it expire when idle and the queue is empty?

It looks like I just need some way to call an arbitrary URL to "notify" it that there are tasks to process but I'm unable to find any documentation on how this can be done.

Answer 1

Use a cron task or a push queue to periodically start the backend. The backend can loop through the tasks (if any) in the pull queue, and then expire.

There isn't a notification system for pull queues, only inspection through queue statistics and empty/non-empty lease results.

Answer 2

First of all you need to decide scalability type you want to use for your module. I think that you should take a look to Basic Scaling (https://developers.google.com/appengine/docs/java/modules/)

Next, to process tasks from pull queue you can use Cron to check queues every several minutes. It will be important to request not basic scaling module, but frontend module, cause cron will start instances. The problem is that you will still need to pay for at least one instance of frontend cause your cron job will not allow it to shutdown.

So the solution could be the following:

1. Start cron every 1 or 5 minutes and call frontend
2. Check queue in frontend and issue URLFetch request to basic scaling module if there are tasks in pull queue
3. Process tasks in queue using basic scaling module

If you use F1 instances for frontend and b2 or greate for other modules it could save you some money.