Elasticsearc + NEST: How to map multiple fields into a .NET Dictionary<string, object>

Question

I'm working on a sample project which will have MongoDB and Elasticsearch. After a long research, I've choose Mongo-Connector to make the sync between MongoDB and Elasticsearch. The sync between them is already working fine.

I've inserted a lot of documents on my MongoDB. Basically the document reference to a "Person". The document has this basic json structure:

{
  "Id": "5a308536-0bd9-47e6-8bdb-438dafd0488c",
  "Name": "Seal",
  "ExtraElements": {
     "DateofBirth": "2001-12-27T03:41:30.333Z",
     "Age": 56,
     "Gender": "F"
  }
}

The json above maps to this .net class

public class Person
{
    public string Id { get; set; }

    public string Name { get; set; }

    public Dictionary<string, object> ExtraElements { get; set; }
}

As you can figure the "ExtraElements" is a dictionary to store N fields. We need this because we don't know how many fields we'll have.

When I insert a json like that in my MongoDB, it gets mapped into a document with this structure:

Person
  Id
  Name
  DateofBirth
  ...
  ...

With Mongo-Connector in place and working, when I've inserted the first document on MongoDB the same document was inserted on Elasticsearch. Since I didn't have a type, it was created on the fly, using dynamic mapping. The mapping created on Elasticsearch is this:

{
  "repository" : {
  "mappings" : {
  "Person" : {
    "properties" : {
      "DateofBirth" : {
         "type" : "date",
         "format" : "dateOptionalTime"
      },
      "Age" : {
        "type" : "long"
      },
      "Gender" : {
        "type": "string"
      }
      "Name" : {
        "type" : "string"
      }
    }
  }
}

} }

As you can see, the ExtraElements fields were created as a "flat" document, or all the fields are in the same "level" meaning that there aren't no nested types.

My question now is: I'm using NEST to query on Elasticsearch, but when I do that, I only get the Id and Name values, the ExtraElements dictionary is null. Did I did something wrong with the mapping stuff on Elasticsearch? Is there a way in c# or NEST to map this?

you can just use a normal C# class instead of a dictionary. You can use GET /your_index/_mapping/your_type to check the actual mapping elasticsearch has for you. You can add fields later with ease. Just do a mapping update. However, changing field name or type will be difficult. — foresightyj

jpgrassi jpgrassi · Accepted Answer · 2015-08-05T18:35:01

I have figured what was the problem.

The problem was the way the data was being saved in my MongoDB. The "ExtraElements" dictionary was being saved inline and not as an object which was my intent. Since Mongo Connector simple replicate data, the same structure was being saved in my Elasticsearch.

After reading intensively the C# Driver documentation, I've came to this solution:

C# Class:

public class Person
{
    public ObjectId Id { get; set; }

    public string Name { get; set; }

    public BsonDocument Metadata { get; set; }

    [BsonIgnore]
    public Dictionary<string, object> ExtraElements { get; set; }
}

Following this: Mixing Static and Dynamic Data I created a BsonDocument and maintained the ExtraElements Dictionary. The dictionary now is marked as "ignored". An example of usage in a API Controller is:

public void Post(Person obj)
{
   obj.Metadata = new BsonDocument(obj.ExtraElements.DeserializeMongoObject())
   personBusiness.Save(obj);
}

The DeserializeMongoObject() is an extension method a colleague created to "handle" multiple levels of object nesting. Without using it, it works for one "level" but if have nested objects inside the ExtraElements, it saves as an array without values.

Using the extension method in conjunction with the BsonDocument I get this result in MongoDB:

And without using it, I get this: (don't know why..)

The extension method code is:

public static Dictionary<string, object> DeserializeMongoObject(this Dictionary<string, dynamic> dictionary)
    {
        if (dictionary.Count > 0)
        {
            for (int i = 0, length = dictionary.Count; i < length; i++)
            {
                if (dictionary[dictionary.ElementAt(i).Key].GetType().ToString() == "Newtonsoft.Json.Linq.JObject")
                {
                    dictionary[dictionary.ElementAt(i).Key] = Newtonsoft.Json.JsonConvert.DeserializeObject<IDictionary<string, object>>(dictionary.ElementAt(i).Value.ToString());
                    dictionary[dictionary.ElementAt(i).Key] = DeserializeMongoObject(dictionary[dictionary.ElementAt(i).Key]);
                }
            }
        }

        return dictionary;
    }

Elasticsearc + NEST: How to map multiple fields into a .NET Dictionary<string, object>

1 Answers