async await WhenAll not waiting

Question

I'm developing WinForms App in VS2013, .NET FW 4.5.1. Here is my reduced code with inline comments about structure:

// Progress object implementing IProgress<MyProgressData>
var progressCallback = new Progress<MyProgressData>();

// listOfMyList is actually List<List<MyObject>>, which contains list of
// list of MyObject's which will be executed as tasks at once.
// For example, this would be sample structure for list of lists:
// List1
//   MyObject1
//   MyObject2
//   MyObject3
// List2
//   MyObject4
//   MyObject5
//   MyObject6
// List1's and List2's objects would be executed as all tasks at once, but List1 and List2 respectively
// would be executed one after another (because of resources usage inside TASK CODE)
foreach (var myItem in listOfMyList)
{
  var myList = myItem.ToList();

  // Create a list of tasks to be executed (20 by default; each taking from 30-60 seconds)
  // Here cs is actually MyObject
  var myTasks = myList.Select(cs => Task.Run(async () =>
  {
    // TASK CODE (using cs as an input object and using "cancellationToken.ThrowIfCancellationRequested();" inside execution to cancel executing if requested)
  }, cancellationToken));

  await Task.WhenAll(myTasks); // Wait for all tasks to finish

  // Report progress to main form (this actually calls an event on my form)
  await Task.Run(() => progressCallback.Report(new MyProgressData() { props }), CancellationToken.None);
}

As you can see, I construct progress object and then I have list of lists. Each item within top-level list should execute in serialized fashion (one after another). Each item's list elements, should execute all at once in a form of a tasks. So far so good, all tasks start and even WhenAll waits for them. Or at least I thought so. I have put logging inside relevant methods, to show me code execution. It turns out that while progress logic (at the bottom) is executing, foreach loop starts executing another batch of tasks, which it shouldn't. Am I missing something here? Does code for progress not block or wait for Report method to finish executing. Maybe I'm missing sth about async/await. With await we make sure that code won't continue until after method is finished? It won't block current thread, but it also won't continue executing? Is it even possible (since its happening, it probably is), for my foreach loop to continue executing while progress reporting is still on the go?

This code resides inside an async method. It's actually called like this (lets assume this method is async MyProblematicMethod()):

while (true)
{
    var result = await MyProblematicMethod();
    if (result.HasToExitWhile)
        break;
}

Every method up from MyProblematicMethod uses await to wait for async methods, and is not called many times.

Unable to reproduce your issue here. Execution does not pass any await operators until the Tasks are done. I would guess your error is somewhere further up the line. Is your method being called multiple times, perhaps? — Glorin Oakenfoot
Edited question to show how its called. It's async all the way and everywhere there is await. It is possible, that I have bug somewhere else, just can't find it for almost a week now. — Jure
This should work. Post the task code. Maybe the lambda body returns. — usr
After digging a little further, I'm going to guess your issue may be with progressCallback.Report(). This method definitely does not wait for the event handlers to finish, as it looks like the handlers run on the thread pool. That means that your progress event handler will likely be executing alongside the next iteration of the loop, which sounds like what you're describing. — Glorin Oakenfoot
It looks like it does run on the main thread in a UI app, which makes sense. Was testing on a console program. In any case, your async method is running on a threadpool thread in this case. It's the same idea. The Task that calls Report() is being awaited, but that Report() call is calling the event handlers on another thread and returning. The event handlers will still run alongside your next loop iteration. — Glorin Oakenfoot

Jure Jure · Accepted Answer · 2015-12-03T16:39:25

Based on Glorin's suggestion that IProgress.Report returns immediately after firing an event handler, I've created exact copy of Progress class, which uses synchronizationContext.Send instead of Post:

public sealed class ProgressEx<T> : IProgress<T>
{
    private readonly SynchronizationContext _synchronizationContext;
    private readonly Action<T> _handler;
    private readonly SendOrPostCallback _invokeHandlers;

    public event EventHandler<T> ProgressChanged;

    public ProgressEx(SynchronizationContext syncContext)
    {
        // From Progress.cs
        //_synchronizationContext = SynchronizationContext.CurrentNoFlow ?? ProgressStatics.DefaultContext;
        _synchronizationContext = syncContext;
        _invokeHandlers = new SendOrPostCallback(InvokeHandlers);
    }

    public ProgressEx(SynchronizationContext syncContext, Action<T> handler)
        : this(syncContext)
    {
        if (handler == null)
            throw new ArgumentNullException("handler");
        _handler = handler;
    }

    private void OnReport(T value)
    {
        // ISSUE: reference to a compiler-generated field
        if (_handler == null && ProgressChanged == null)
            return;
        _synchronizationContext.Send(_invokeHandlers, (object)value);
    }

    void IProgress<T>.Report(T value)
    {
        OnReport(value);
    }

    private void InvokeHandlers(object state)
    {
        T e = (T)state;
        Action<T> action = _handler;
        // ISSUE: reference to a compiler-generated field
        EventHandler<T> eventHandler = ProgressChanged;
        if (action != null)
            action(e);
        if (eventHandler == null)
            return;
        eventHandler((object)this, e);
    }
}

This means that ProgressEx.Report will wait for method to finish, before returning. Maybe not the best solution in all situations, but it worked for me in this case.

To call it, just create ProgressEx with SynchronizationContext.Current as a parameter to constructor. However, it must be created in UI thread, so the right SynchronizationContext gets passed in. E.g. new ProgressEx<MyDataObject>(SynchronizationContext.Current)

async await WhenAll not waiting

1 Answers