GAE push queue tasks are getting blocked

Question

Got an issue with GAE push queue tasks running in Backends. Some of the tasks are running indefinitely and blocking other tasks from running.

Attaching Logs screenshot obtained after backend restarted - http://snag.gy/3xeyE.jpg

Push Queue task configuration is as follows

Max rate 5/s | Enforced rate 5/s | Bucket size 5.0 | Max Concurrent 10 | Retry limit 1

The task sends request to Backend servlet in which tasks are leased from other pull queue and executed until backend stops or pull queue lease returns empty in infinite loop.

We are careful that the infinite loop breaks if no tasks are leased. There are no logs given under these requests to know what these processes are running.

Few observations:

'cpu_ms' is 0 for most of these processes

No 'pending_ms' too for most of them

Other tasks in the queue are blocked until these tasks complete.

This is very serious issue for us.

Please help me.

Answer 1

I suggest you to try the following:

1. Unpause the queue which has tasks stuck inside;

2. Delete all the tasks within the Queue associated with the specific jobs which is stuck;

3. Delete all the kinds associated with the specific jobs or tasks in the Datastore;

4. Re-start the specific tasks in the queue.