Run blocking and unblocking tasks together with asyncio

Question

I want to run blocking and unblocking tasks together asynchronously. Obviously that it is necessary to use run_in_executor method for blocking tasks from asyncio. Here is my sample code:

import asyncio
import concurrent.futures
import datetime
import time


def blocking():
    print("Enter to blocking()", datetime.datetime.now().time())
    time.sleep(2)
    print("Exited from blocking()", datetime.datetime.now().time())


async def waiter():
    print("Enter to waiter()", datetime.datetime.now().time())
    await asyncio.sleep(3)
    print("Exit from waiter()", datetime.datetime.now().time())


async def asynchronous(loop):
    print("Create tasks", datetime.datetime.now().time())
    task_1 = asyncio.create_task(waiter())

    executor = concurrent.futures.ThreadPoolExecutor(max_workers=3)
    task_2 = loop.run_in_executor(executor, blocking)

    tasks = [task_1, task_2]
    print("Tasks are created", datetime.datetime.now().time())
    await asyncio.wait(tasks)


if __name__ == "__main__":
    try:
        loop = asyncio.get_event_loop()
        loop.run_until_complete(asynchronous(loop))
    except (OSError) as exc:
        sys.exit('Exception: ' + str(exc))

Should I use the same event loop for blocking task in run_in_executor, or it is necessary to use another one? What should I change in my code to make it work asynchronously? Thanks

In addition to what the answer says, note that you don't need to create a new thread pool for each run. The point of a thread pool is to reuse the threads for efficiency. You can simply pass None as the first argument to run_in_executor and have it use the thread pool created by asyncio for the event loop. — user4815162342
oh, I see. But if I will not use separate threads from custom thread pull, it will block thread with event loop, isn't it? — d.golov
I'm not saying you should use a thread pool, I'm just saying you don't need to create your own thread pool. Passing None to run_in_executor will use asyncio's own thread pool which exists for this very purpose. That thread pool still has its own threads and won't block the event pool thread. — user4815162342
I see, now. I just was confused with this phrase from official documentation: "The loop.run_in_executor() method can be used with a concurrent.futures.ThreadPoolExecutor to execute blocking code in a different OS thread without blocking the OS thread that the event loop runs in." I thought that by default there is only one thread with event loop, and using your own thread pool you will avoid block of main thread with event loop — d.golov
You thought correctly, there is indeed only one thread that runs the event loop. This thread pool is a helper thing provided for situations such as yours, to avoid everyone creating their own thread pool and the number of threads growing without bounds. So it's true that you have to use a thread pool, but it doesn't have to be your own - the thread pool provided by asyncio (which is distinct from the thread that runs the event loop) is perfectly fine. — user4815162342

Dunes Dunes · Accepted Answer · 2020-09-02T11:11:04

You must use the same loop. The loop delegates to the executor, which runs tasks is separate threads to the event loop. So you don't have to worry about your blocking tasks blocking the event loop. If you use a separate loop, your async functions from the event loop will not be able to await the results of blocking the functions run in the new loop.

The event loop manages this by creating a future to represent the executor task. It then runs the blocking task in one of the executors threads, and when the executor task returns the result of the future is set and control returned to awaiting function in the event loop (if any).

Run blocking and unblocking tasks together with asyncio

1 Answers