how to manage an NDC-like log4net stack with async/await methods? (per-Task stack?)

Question

In a normal/synchronous/single-threaded console app, NDC.Push works fine for managing the 'current item' (potentially at multiple levels of nesting, but just 1 level for this example).

For example:

private static ILog s_logger = LogManager.GetLogger("Program");

static void Main(string[] args)
{
    BasicConfigurator.Configure();

    DoSomeWork("chunk 1");
    DoSomeWork("chunk 2");
    DoSomeWork("chunk 3");
}

static void DoSomeWork(string chunkName)
{
    using (NDC.Push(chunkName))
    {
        s_logger.Info("Starting to do work");
        Thread.Sleep(5000);
        s_logger.Info("Finishing work");
    }
}

This will result in the expect log output, showing a 'chunk X' NDC entry just to the right of 'Program' (the default pattern for the basic configurator)

232 [9] INFO Program chunk 1 - Starting to do work

5279 [9] INFO Program chunk 1 - Finishing work

5279 [9] INFO Program chunk 2 - Starting to do work

10292 [9] INFO Program chunk 2 - Finishing work

10292 [9] INFO Program chunk 3 - Starting to do work

15299 [9] INFO Program chunk 3 - Finishing work

However, I can't figure out how to maintain that using 'normal' async methods.

For instance, trying to do this:

private static ILog s_logger = LogManager.GetLogger("Program");

static void Main(string[] args)
{
    BasicConfigurator.Configure();

    var task1 = DoSomeWork("chunk 1");
    var task2 = DoSomeWork("chunk 2");
    var task3 = DoSomeWork("chunk 3");

    Task.WaitAll(task1, task2, task3);
}

static async Task DoSomeWork(string chunkName)
{
    using (log4net.LogicalThreadContext.Stacks["NDC"].Push(chunkName))
    //using (log4net.ThreadContext.Stacks["NDC"].Push(chunkName))
    {
        s_logger.Info("Starting to do work");
        await Task.Delay(5000);
        s_logger.Info("Finishing work");
    }
}

Shows them all starting "normally", but when the task completes on a different thread, the stack is lost (I was hoping the log4net.LogicalThreadContext would be TPL-'aware' I guess).

234 [10] INFO Program chunk 1 - Starting to do work

265 [10] INFO Program chunk 2 - Starting to do work

265 [10] INFO Program chunk 3 - Starting to do work

5280 [7] INFO Program (null) - Finishing work

5280 [12] INFO Program (null) - Finishing work

5280 [12] INFO Program (null) - Finishing work

Outside of adding a new TaskContext (or the like) to log4net, is there a way of tracking this kind of activity?

The goal is really to do so with the async/await syntax sugar - either forcing some kind of thread affinity or doing things like keeping a concurrent dictionary around keyed by task are likely workable options, but I'm trying to keep as close to the synchronous version of the code as possible. :)

FYI, I recently discovered that Microsoft fixed CallContext in .NET 4.5 RTW to work with async. So log4net's NDC and other solutions using Logical*Data will work as expected with async methods (on .NET 4.5 only). — Stephen Cleary
I'm not sure it's a problem coming from the logical thread context. The log4net implementation looks wrong to me because parent and child thread share the same stack. The child should receive a clone of the parent stack so that if the parent modifies the stack it doesn't perturb the child... — dou bret

Stephen Cleary Stephen Cleary · Accepted Answer · 2012-03-20T14:51:42

There isn't a good story for async logical call contexts right now.

CallContext can't be used for this. The logical CallContext doesn't understand how async methods return early and resume later, so it will not always work correctly for code that uses simple parallelism such as Task.WhenAll.

Update: CallContext was updated in .NET 4.5 RTW to correctly work with async methods.

I looked into log4net; LogicalThreadContext is documented as using CallContext, but there was a bug that made it use the non-logical contexts (fixed in their SVN on Feb 2, 2012; the current 1.2.11 release does not include that fix). Even when it's fixed, though, it'll still not work with async (because the logical CallContext doesn't work with async).

When I need an async logical call context, I make a class that contains the context data and keep all of my async methods in a functional style as instance members of that class. This is certainly not an ideal solution, but it's a dirty hack that works.

In the meantime, please upvote the suggestion that Microsoft provide some mechanism for this.

P.S. A concurrent dictionary keyed by Task won't work, because async methods are not necessarily running tasks (i.e., in your example code, at the using statement, Task.CurrentId would be null because there is no task actually executing at that point).

And thread affinity has its own problems, too. You actually end up needing a separate thread for each independent asynchronous operation. Bye-bye, scalability...

how to manage an NDC-like log4net stack with async/await methods? (per-Task stack?)

1 Answers