How to have different log types using Serilog and ElasticSearch

Question

I am currently trying to change our system configuration to work with Serilog (instead of working with FileBeat as a shipper to LogStash)

We are also working with the log type field (which is easy to configure in the FileBeat config file) in our various queries and for indexing out logs on Elastic.

The problem is that when using Serilog we get the default type logevent, and I didn't find where I can configure it. I want to have an option to determine a specific log type per Serilog instance. At the moment I have the default type for all of my logs.

my Serilop Config is:

        var path = GetLogPath();
        var logger = new LoggerConfiguration()
            .MinimumLevel.Information()
            .Enrich.WithMachineName()
            .Enrich.WithProperty("RequestId", Guid.NewGuid())
            .WriteTo.RollingFile(
                pathFormat: path,
                outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss} [{Level:u4}] [{RequestId}] {Message}{NewLine}{Exception}", buffered: false, shared: true);
        logger.WriteTo.Elasticsearch(
                new ElasticsearchSinkOptions(new Uri(this.configurationService.ElasticSearchUrl())));

How can I change the log type?

EDIT

After some investigation, I found out I suppose to change the typeName filed in the LoggerConfiguration and seem that I can do so only through the AppConfig file, which again event if I will change it, the change will affect all the logger instances.

Do I miss something?

more info about what type means to you. In general, in Serilog, there is a {SourceContext} property you can put in as a token into the emit format if you mean the type associated with the logger. if you mean an id for the message template, github.com/serilog/serilog-formatting-compact will give you clues. (I know zero about the ES sink (obviously) but if you emit json, you can normally use the format to guid what to include) — Ruben Bartelink
thanx Ruben, I did some reading about this property and it indeed sounds like good practice. I'm not sure if it will come hand to hand with my problem that related for the log indexing in the Eleastic — Green

Hooman Bahreini Hooman Bahreini · Accepted Answer · 2018-05-13T09:33:34

Updated Answer

To add Properties and values to your logger, you can use Contextual logging and Enrichment

Contextual Logger

The simplest and most direct way to attach contextual properties to a log event

First initialized your logger:

Log.Logger = new LoggerConfiguration().ReadFrom.AppSettings().CreateLogger();

Then you can create your contextual logger:

// adding Log Context
var StudentLogger = Log.Logger.ForContext<Student>();

StudentLogger.Error(/* log message */);

Or you can use correlation log entries:

// correlation Log Entries
var orderId = "some value";
var corrLog = Log.Logger.ForContext("orderId", orderId)

corrLog.Error(/* log message */);

Enrichment

In some cases we would like every event created by a logger to carry the same, fixed, property value. The Application example is one of these.

Serilog provides Enrich.WithProperty() at the level of a LoggerConfiguration for this:

Log.Logger = new LoggerConfiguration()
    .Enrich.WithProperty("Application", "e-Commerce")
    .Enrich.WithProperty("Environment", ConfigurationManager.AppSettings["Environment"])
    // Other logger configuration

Original Answer

There are two ways to configure Serilog:

Using API (need serilog.sinks.elasticsearch package):

var loggerConfig = new LoggerConfiguration()
    .MinimumLevel.Debug()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200") ){
         AutoRegisterTemplate = true,
 });
var logger = loggerConfig.CreateLogger();

Serilog Documentation

Using configuration from AppSettings (need Serilog.Settings.AppSettings in addition to serilog.sinks.elasticsearch)

This way, you put all of your setting in AppSetting file, e.g.

<appSettings>
    <add key="serilog:using" value="Serilog.Sinks.Elasticsearch"/>
    <add key="serilog:write-to:Elasticsearch.nodeUris" value="http://localhost:9200;http://remotehost:9200"/>
    <add key="serilog:write-to:Elasticsearch.indexFormat" value="custom-index-{0:yyyy.MM}"/>
    <add key="serilog:write-to:Elasticsearch.templateName" value="myCustomTemplate"/>
  </appSettings>

And tell serilog to read the config from appSettigns

Log.Logger = new LoggerConfiguration()
  .ReadFrom.AppSettings()
  ... // Other configuration here, then
  .CreateLogger()

See: AppSetting and ElasticSearch Configure Sink

I am not sure which log event type you are referring to? In my case, I pass the object type while logging the errors:

catch (Exception ex)
{
    Logger.Error(ex, string.Format("Exception occured in Controller: {0}, Action: Post.", this.GetType()), this.GetType());

How to have different log types using Serilog and ElasticSearch

3 Answers

Updated Answer

Original Answer