How do we extract Serilog .ForContext() objects logged to Elasticsearch, using ElasticClient (NEST)?

Question

We are looking to better centralize, structure and search our logs using the ELK stack. We are currently logging to a database table, and we have an EventLog POCO that captures the fields required to populate those rows. Our first stab at getting this data into Elasticsearch was to simply log the events via Serilog like this:

EventLog eventLog = ...;
log.ForContext("EventLog", eventLog, true).Write(eventLog.EventMessage);

This results in a Serilog "logevent"-type document, with a "field" that contains our custom structure, and gets us most of what we want:

{
  "_index": "logstash-2019.03.14",
  "_type": "logevent",
  "_id": "p_amfGkBaphWeJXGjjSW",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-03-14T09:41:21.6924251-05:00",
    "level": "Information",
    "messageTemplate": "blah",
    "message": "blah",
    "fields": {
      "EventLog": {
        "_typeTag": "EventLog",
        "EventMessage": "blah",
        "EventId": 3112,
...

Kibana allows us to do ad-hoc searches of our logs on fields in our custom structure, and we're fairly happy already.

Now we want to search our logs programmatically, and we're looking at NEST for that, but we can't quite figure out the right incantations. ElasticClient.Search<T> seems to be what we need, we're not sure what T needs to be, nor how NEST interprets the search criteria or the returned data.

What we've tried:

client.Search<EventLog>();   // returns zero results
client.Search<Serilog.Events.LogEvent>();   // throws: Error converting string to Serilog.Events.MessageTemplate
client.Search<EventLog>(s => s.Type("logevent")); // returns results!, but the EventLog objects are all empty (properties all have default values)

The only call that seems to get us anywhere close is:

client.Search<Object>(s => s.Type("logevent")); // returns the raw JSON source of each log entry

So, we feel like we're missing something simple and fundamental here. The NEST docs are all predicated on a Project type that is indexed directly into Elasticsearch, but what to we need to do to be able to extract and query over our custom structures that we add to the log using Serilog's ForContext?

UPDATE: It seems like we might need to construct a separate set of POCOs that replicate the structure of a Serilog event, eg:

        private class DummyFields
        {
            public EventLog EventLog { get; set; }
        }

        private class Dummy
        {
            public DummyFields Fields { get; set; }
        }

Querying on Dummy in NEST does indeed give us access to our inner structure, but it seems like there should be a better way to do this...

UPDATE 2: We're OK with implementing a POCO just for matching query responses, but it now seems there's some other problem with matching on fields inside our nested structure:

client.Search<Dummy>(s => s
    .AllTypes()
    .Query(q => q
        .Match(t => t
            .Field(f => f.Message).Query("blah"); // returns some results

client.Search<Dummy>(s => s
    .AllTypes()
    .Query(q => q
        .Match(t => t
            .Field(f => f.Fields.EventLog.EventMessage).Query("blah"); // returns zero results

Why doesn't the second query "see" our nested event message?

UPDATE 3:

Curiously, this also works:

client.Search<Dummy>(s => s
    .AllTypes()
    .Query(q => q
        .Match(t => t
            .Field(f => new Field("fields.EventLog.EventMessage")).Query("blah");

So it looks like NEST just isn't matching the type-safe version of the nested field. We'd obviously prefer not to have to rely on magic strings here...

UPDATE 4:

I enabled query logging in ES and saw that there's a case difference between the literal and expression queries. The literal query (with PascalCased properties) is seen as Pascal cased by ES and finds the documents, while the type-safe expression gets converted to camelCase ("fields.eventLog.eventMessage"), and does not match anything. Sure enough, if I query with a literal camelCased field path, I get nothing back also.

How do I tell NEST to use the right case sensitivity?

Russ Cam Russ Cam · Accepted Answer · 2019-03-15T07:05:48

The original JSON document sent to Elasticsearch is contained within the _source property of each hit in the search response

{
    "@timestamp": "2019-03-14T09:41:21.6924251-05:00",
    "level": "Information",
    "messageTemplate": "blah",
    "message": "blah",
    "fields": {
      "EventLog": {
        "_typeTag": "EventLog",

The "EventLog" that I believe you're trying to get to is at the JSON path "fields.EventLog", so any CLR POCO that would map to this and into which the JSON would be deserialized would need to conform to this structure, as your Dummy type does above.

There would be two approaches that you could take to change this:

serialize "EventLog" as the entire JSON document in _source. You would then be able to map it directly to your "EventLog" type.
Implement a custom serializer that deserializes only "fields.EventLog" to a given EventLog POCO type. You'd be able to do this with JsonNetSerializer and implement a custom JsonConverter for EventLog type. There are two complexities to doing this though; firstly, this would only operate as a read model because when wishing to serialize again, the resulting JSON will only be the JSON for EventLog, and secondly, using JsonNetSerializer has a performance overhead compared to the internal serializer used by NEST.

How do we extract Serilog .ForContext() objects logged to Elasticsearch, using ElasticClient (NEST)?

1 Answers