Best practices for using NLOG with a multi-user based system?

Question

We need to separate logs generated by a REST web-service on a user-specific basis and eventually import these logs into an aggregation framework like Datadogs.com.

There are several ways to approach this and I’m interested in getting feedback before selecting an approach.

The basics would go something like this:

Rest service is called with UserID as an argument.
Set the nlog LogManager.Configuration.Variable["userid"] = userid, or use MappedDiagnosticContext.
Write the log normally until the REST call completes.

Depending on the stage

For development, use the NLOG File Logger, so the developer can simply “tail” the log file. Use the variable in the nlog filename target as
<target filename="/path/file-${var:userid}.log"/>. Or use the, it in the nlog target as: layout="${var:userid}-${OtherLayout}", and have developers do a tail -f masterFile.log | grep USERID.
In production switch to using the nlog JsonLayout, so a system like DataDog can read Time, Threadid, Userid and message data. Use the JsonFormat and specify a userid attributes. I see that the JsonFormat has supports for MappedDiagnosticsLogicalContext but I would prefer the simplicity of the ${var:xxx} format to specify the value.

Problems:

I’ve used the MappedDiagnosticContext in the past and then specified it in the target filename. I just attempted to use the ${var} approach and it did not appear to work??

Concerns:

I like the filename target approach for development. It should work well with 10-100 of users but not scale with 1000's of users. Clearly with 1000’s of users we would need to close the log file after each line is written, as we don’t want to keep 1000’s of files open.

A major concern is threading. It’s possible that each webservice is called multiple times by different users and all approaches require Nlog MappedDiagnosticContext or ${var} capability’s to be thread safe. Is it? Any issues to consider?

Eventually we would like to introduce some structured logging into the system, but the majority of the code base was built using standard logging techniques. If the objects being logged in the structured logging included userid then much of this complexity could be avoided, but that would require a lot of work rewriting for structured logging.

Clearly, there is a lot to think about and I know I’m not the first to ponder this.

Your input will be appreciated.

Just use NLog.MappedDiagnosticsLogicalContext.Set("userid", "someValue") together with ${mdlc:item=userid} where needed. See also github.com/NLog/NLog/wiki/MDLC-Layout-Renderer — Rolf Kristensen
You should avoid changing LogManager.Configuration.Variable at runtime, they are global for all concurrent requests, and might get lost during configuration-reload (If autoreload configured). — Rolf Kristensen
I did it and it works.. Question: do you know if this is thread safe? — mbalsam
Yes MappedDiagnosticsLogicalContext is threadsafe, which I have now explained in my answer. — Rolf Kristensen

Rolf Kristensen Rolf Kristensen · Accepted Answer · 2018-07-27T06:49:35

Just use NLog.MappedDiagnosticsLogicalContext.Set("userid", "someValue") together with ${mdlc:item=userid} where needed. See also http://github.com/NLog/NLog/wiki/MDLC-Layout-Renderer

MappedDiagnosticsLogicalContext uses CallContext (and AsyncLocal on NetCore) which are thread-safe. Settings will also support async Task and follow to the chained tasks, but if scheduling a Time-callback based on a user-request, then the Timer-callback will not see the userid.

You should avoid changing LogManager.Configuration.Variable at runtime, they are global for all concurrent requests, and might get lost during configuration-reload (If autoreload configured).

Best practices for using NLOG with a multi-user based system?

1 Answers