NEST dotnet core elastic not properly indexing

Question

I'm having trouble with NEST and elastic logger in dot net core (v3 preview6).

I have a log class note the data attribute, I would like this to accept any json I throw at it, which from the API/route perspective works fine, I can populate the attribute, however it doesn't correctly transfer to elastic search via NEST. I have tried sending this as a JObject JToken and System.Text.Json.JsonElement:

    public class Log
    {

        public System.DateTime timestamp { get; set; }
        public System.Text.Json.JsonElement data {get; set;}
    }

Then I use my Elastic Client as such:

var response = client.Index(log, idx => idx.Index("log"));

I can't get it to turn my dynamic json into json in elastic search regardless of the type. What am I missing?

Russ Cam Russ Cam · Accepted Answer · 2019-07-25T04:54:19

The problem here is that NEST 6.x+ (I'll use NEST 7.1.0 in these examples) do not know how to serialize JObject, JToken or JsonElement, or more precisely, the default internal serializer does not know how to serialize them.

Take JObject for example

public class Log
{
    public DateTime timestamp { get; set; }
    public JObject data { get; set; }
}

and serialized

var client = new ElasticClient();

var log = new Log
{
    timestamp = new DateTime(2019,7,25, 0,0,0, DateTimeKind.Utc),
    data = new JObject 
    {
        { "foo", "bar" },
        { "baz", new JArray(1, 2) },
    }
};

client.Index(log, i => i.Index("log"));

yields

POST http://localhost:9200/log/_doc
{"timestamp":"2019-07-25T00:00:00Z","data":[[[]],[[[],[]]]]}

"data" is serialized as an array of arrays. Json.NET has special handling for JObject to serialize it as an object, but for any other serializer, it's going to use conventions based on the object type or what interfaces the objects implements, and an order in which these might be reflected over and determined. If you want to use JObject with NEST 6.x or 7.x, you can use the NEST.JsonNetSerializer nuget package and configure it as the serializer for your POCOs. Note that there is some performance overhead to using JsonNetSerializer, at the expense of flexibility.

If you don't want to go that route, You can use DynamicDictionary in NEST 7.x, which is a dictionary that supports dynamic access

var client = new ElasticClient();

var log = new Log
{
    timestamp = new DateTime(2019,7,25, 0,0,0, DateTimeKind.Utc),
    data = new DynamicDictionary 
    {
        { "foo", new DynamicValue("bar") },
        { "baz", new DynamicValue(new [] { 1, 2 }) },
    }
};

client.Index(log, i => i.Index("log"));

which serializes to

POST http://localhost:9200/log/_doc?pretty=true 
{"timestamp":"2019-07-25T00:00:00Z","data":{"foo":"bar","baz":[1,2]}}

NEST dotnet core elastic not properly indexing

1 Answers