How does async-await "save threads"?

5
votes

I understand that with threadless async there are more threads available to service inputs (e.g. a HTTP request), but I don't understand how that doesn't potentially cause cause thread starvation when the async operations complete and a thread is needed to run their continuation.

Let's say we only have 3 threads

Thread 1 | 
Thread 2 |
Thread 3 |

and they get blocked on long-running operations that require threads (e.g. make database query on separate db server)

Thread 1 | --- | Start servicing request 1 | Long-running operation .................. |
Thread 2 | ------------ | Start servicing request 2 | Long-running operation ......... |
Thread 3 | ------------------- | Start servicing request 3 | Long-running operation ...|
               |
              request 1
                        |
                      request 2
                                |
                              request 3
                                               |
                                           request 4 - BOOM!!!!

With async-await you can make this like

Thread 1 | --- | Start servicing request 1 | --- | Start servicing request 4 | ----- |
Thread 2 | ------------ | Start servicing request 2 | ------------------------------ |
Thread 3 | ------------------- | Start servicing request 3 | ----------------------- |
               |
              request 1
                        |
                      request 2
                                |
                              request 3
                                                 |
                                           request 4 - OK

However, this seems to me like it could result in a surplus of async operations that are "in-flight" and if too many finish at the same time then there are no threads available to run their continuation. 

Thread 1 | --- | Start servicing request 1 | --- | Start servicing request 4 | ----- |
Thread 2 | ------------ | Start servicing request 2 | ------------------------------ |
Thread 3 | ------------------- | Start servicing request 3 | ----------------------- |
               |
              request 1
                        |
                      request 2
                                |
                              request 3
                                                 |
                                           request 4 - OK   
                                                      | longer-running operation 1 completes - BOOM!!!!
2
c#.netmultithreadingasynchronousasync-await
Proper use of async and await implies that no threads are ever created, so this question is either based on a false premise or missing details - Camilo Terevinto
A typical async await operation does not create a new thread. Rather, it signs onto the remainder of the method as a continuation, where its execution is resumed later, on the same thread. This article describes how it works in detail; it even includes a diagram. - Robert Harvey
You only really need multiple threads when you have CPU bound work. A lot of Multitasking work is anything but CPU bound. Waiting for User input or network returns - that does not need a lot of cycles. But for historical reasons it was for a long time easier to always use Threads for Multitasking. So any way we can do Multitasking without having to resort to Multithreading uselessly saves threads. Async...await is such a way. - Christopher
If you decompile the code with async/await you will see that it is actually a clever use of a goto. Basically the thread does no work and the execution is then passed to the next statement when IO operation finishes. Kind of like event handling. - zaitsman
From what I understand, when using old fashioned threading, management of parallelization (switching thread) is mostly pushed down to the OS, which is costly. When using async-await / Task / async or mailbox in F#, management (switching task or async) is done mostly in the Framework/App, which is not costly. - Bent Tranberg

2 Answers

4
votes

Suppose you have web application which handles a request with a very common flow:

  • Preprocess request parameters
  • Perform some IO
  • Post process IO results and return back to client

IO in this case can be database query, socket read\write, file read\write and so on.

For an example of IO let's take file reading and some arbitrary but realistic timings:

  1. Preprocessing of request parameters (validation etc) takes 1ms
  2. File reading (IO) takes 300 ms
  3. Post processing takes 1ms

Now suppose 100 requests come in with interval of 1ms. How many threads you will need to handle those requests without delay with synchronous processing like this?

public IActionResult GetSomeFile(RequestParameters p) {
    string filePath = Preprocess(p);
    var data = System.IO.File.ReadAllBytes(filePath);
    return PostProcess(data);
}

Well, 100 threads obviously. Since file read takes 300ms in our example, when 100th request comes in - previous 99 are busy blocked by file reading.

Now let's "use async await":

public async Task<IActionResult> GetSomeFileAsync(RequestParameters p) {
    string filePath = Preprocess(p);
    byte[] data;
    using (var fs = System.IO.File.OpenRead(filePath)) {
        data = new byte[fs.Length];
        await fs.ReadAsync(data, 0, data.Length);
    }
    return PostProcess(data);
}

How many threads are needed now to handle 100 requests without delay? Still 100. That's because file can be opened in "synchornous" and "asynchronous" modes, and by default it opens in "synchronous". That means even though you are using ReadAsync - underlying IO is not asynchronous and some thread from a thread pool is blocked waiting for result. Did we achieve anything useful by doing that? In context of web applicaiton - not at all.

Now let's open file in "asynchronous" mode:

public async Task<IActionResult> GetSomeFileReallyAsync(RequestParameters p) {
    string filePath = Preprocess(p);
    byte[] data;
    using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, FileOptions.Asynchronous)) {
        data = new byte[fs.Length];
        await fs.ReadAsync(data, 0, data.Length);
    }

    return PostProcess(data);
}

How many threads we need now? Now 1 thread is enough, in theory. When you open file in "asynchronous" mode - reads and writes will utilize (on windows) windows overlapped IO.

In simplified terms it works like this: there is a queue-like object (IO completion port) where OS can post notifications about completions of certain IO operations. .NET thread pool registers one such IO completion port. There is only one thread pool per .NET application, so there is one IO completion port.

When file is opened in "asynchronous" mode - it binds its file handle to this IO completion port. Now when you do ReadAsync, while actual read is performed - no dedicated (for this specific operation) thread is blocked waiting for that read to complete. When OS notify .NET completion port that IO for this file handle has completed - .NET thread pool executes continuation on thread pool thread.

Now let's see how processing of 100 requests with 1ms interval can go in our scenario:

  • Request 1 goes in, we grab thread from a pool to execute 1ms pre-processing step. Then thread performs asynchronous read. It doesn't need to block waiting for completion, so it returns to the pool.

  • Request 2 goes in. We have a thread in a pool already which just completed pre-processing of request 1. We don't need an additional thread - we can use that one again.

  • Same is true for all 100 requests.

  • After handling pre-processing of 100 requests, there are 200ms until first IO completion will arrive, in which our 1 thread can do even more useful work.

  • IO completion events start to arrive - but our post-processing step is also very short (1ms). Only one thread again can handle them all.

This is an idealized scenario of course, but it shows how not "async await" but specifically asynchronous IO can help you to "save threads".

What if our post-processing step is not short but we instead decided to do heavy CPU bound work in it? Well, that will cause thread pool starvation. Thread pool will create new threads without delay, until it reaches configurable "low watermark" (which you can obtain via ThreadPool.GetMinThreads() and change via ThreadPool.SetMinThreads()). After that amount of threads is reached - thread pool will try to wait for one of the busy threads to become free. It will not wait forever of course, usually it will wait for 0.5-1 seconds and if no thread become free - it will create a new one. Still, that delay might slow your web application quite a bit in heavy load scenarios. So don't violate thread pool assumptions - don't run long CPU-bound work on thread pool threads.

0
votes

The fact is, the notion that async/await "saves threads" is a mixture of truth and bullshit. It is true that it doesn't generally involve creating more threads just to service a particular task, but it happily glosses over the fact that under the covers there are a number of threads waiting for events on completion ports, which are created by the runtime. The number of completion port threads is about the number of processor cores in the system. So, on a system with eight processor cores, there are around eight threads waiting for IO completion events. In an application that goes nuts with async IO, that's great, but in an application that doesn't do so much IO, they're mostly just sitting there eating resources, not "saving threads" by any stretch of the imagination.

When an async IO operation completes, one of those threads will "wake up" and eventually call the continuation on whatever task is relevant. If all of the completion threads are busy executing continuations (perhaps because the developer has made the mistake of doing a lot of CPU intensive work in continuations) when another IO operation completes, that completion will not be handled until one of the completion threads is freed up and is able handle it. This is what is referred to as "thread starvation", and it is why it is recommended to start more threads than the number of processor cores in applications that make heavy use of async IO.

The problem with .NET and async IO and the blanket notion that async IO "saves threads", is that many developers don't understand what's actually happening under the covers, and mis-using the async/await pattern in ways that can starve the completion thread pool is all too easy.

In any case, "threadless" is not a term that makes any sense whatsoever here.