Python Asyncio how to run Tasks/Coroutines Consecutively from a Queue

Question

In another program I have made that utilises asyncio I am having to following problem:

in one function, two coroutines are awaited (on separate lines), however the first takes some time, before the first coroutine finishes, another function is called which awaits another coroutine, since this is awaited before the first finishes, the second coroutine of the first function has not been awaited yet, so the second function is run between the two coroutines of the first

My solution was to make some sort of global queue that coroutines would be added to so the run time wouldn't affect the order, I made a small test, but can't get it to work properly.

I tried using a while not co.done() loop, but it seems to only resolve after the task finishes

I tried doing it outside a loop, but for some reason it completes all of the tasks with just one q.get()

import asyncio
from time import gmtime, strftime 

async def task(name, time, queue=None):

    print(strftime("%H:%M:%S", gmtime()))
    await asyncio.sleep(time)
    print("task %s done"%(name))
    print(strftime("%H:%M:%S", gmtime()))

    if queue:

        queue.task_done()

async def main():

    q = asyncio.Queue()
    await q.put(asyncio.create_task(task("A", 3, q)))
    await q.put(asyncio.create_task(task("B", 1, q)))
    await q.put(asyncio.create_task(task("C", 1, q)))

    for i in range(3):

        co = await q.get()
        done, pending = await asyncio.wait({co})

asyncio.run(main())

I was expecting it to complete all the tasks one after another, so the total time would be 5 seconds. But it runs them concurrently in 3 seconds with B and C finishing first at the same time, then A 2 seconds later.

Tobotimus Tobotimus · Accepted Answer · 2019-08-13T01:45:24

The reason your example is running all three tasks concurrently is because asyncio.create_task() schedules the coroutines to run even before you pop them off the queue and await them. If I'm understanding correctly, the calls to asyncio.create_task() here are unnecessary (as is the call to asyncio.wait()). You could simply refactor your main() function like this, and it will run the tasks in FIFO order, and only one task will be run at a time:

async def main():

    q = asyncio.Queue()
    await q.put(task("A", 3, q))
    await q.put(task("B", 1, q))
    await q.put(task("C", 1, q))

    for i in range(3):

        co = await q.get()
        await co

Python Asyncio how to run Tasks/Coroutines Consecutively from a Queue

1 Answers