Google application engine task queue - allow only one item

1
votes

I have this strange problem with push queue.

I need only one task to run at a time from this queue. I have configured a push queue with max-concurrent-requests set to 1. But when I submit tasks to the queue, they end up running in parallel. I had to decrease the rate to 1 per minute. But what happens if my task takes more than 1 min? The two of them will run in parallel.

<queue>
    <name>winCoinsCalculatorQueue</name>
    <max-concurrent-requests>1</max-concurrent-requests>
    <rate>1/m</rate>
</queue>

I also tried 1 but it still seems to process multiple items in parallel.

Anyone knows how to make the queue allow processing only one task at a time?

Arun

2
google-app-engine
can you elaborate why you think multiple tasks running in parallel when you set max concurrent to 1? - marcadian
When I put a debug on my servlet which runs the task, I see two threads stuck on the debug. If I submit three tasks, I see three threads. Its totally ignoring the max-concurrent-requests - arun_gopalan

2 Answers

0
votes

That a fine way to prevent tasks from running concurrently. One of your comments suggests you have only tested on the development server. The dev server does not respect some of the task queue settings. Try it in production, it should work.

You could verify this by making sure your tasks take a long time to complete, then queueing a few. Go to the console for your project, look under App Engine -> Task Queues, and click on your queue's name. You'll see there how many are currently running.

0
votes

First of all, even if you implement a mechanism for running only one task at a time, but the interval between the tasks is shorter than the task run time, you're going to get your task queue overpopulated. That said, I see two possible solutions.

Solution one - action token. Make a datastore entry that will tell if there's a task running. Now, when each task starts to run, it checks the DB, to see if another task is running, if another task is running, enqueue current task to try again in N seconds (use the countdown argument: taskqueue.add(url='/tasks/your_task', countdown=delay_in_seconds, params=params)) (please note, you'll have to find a way to deal with a race condition here)

Solution two - Use pull queues and a cron job that runs every N seconds, again, with action token. Every time the corn job runs, it checks if there's a task running, if it is, the cron doesn't do anything. If no task is running it sets the running flag in the DB. When the task finishes it will unset it.