MongoDB-MongoRiver-ElasticSearch-NEST

Question

I have setup MongoDB database with Enron mail collection. MongoRiver is used to bridge between Mongo & ElasticSearch. I can access the Mongo Data via REST on ElasticSearch.

While configuring MongoRiver, I have already created an index "MongoIndex"

Issue: When I use NEST(.NET C#) to connect/fetch the data from ES, it returns 0 document whereas the same query returns result on REST.

MongoDB Details - DB: Enron Collection: Messages

ES Details- Index: MongoIndex Type: Messages

Single document format for a message in the collection -

{
    "_id" : ObjectId("4f2ad4c4d1e2d3f15a000000"),
    "body" : "Here is our forecast\n\n ",
    "subFolder" : "allen-p/_sent_mail",
    "mailbox" : "maildir",
    "filename" : "1.",
    "headers" : {
        "X-cc" : "",
        "From" : "[email protected]",
        "Subject" : "",
        "X-Folder" : "\\Phillip_Allen_Jan2002_1\\Allen, Phillip K.\\'Sent Mail",
        "Content-Transfer-Encoding" : "7bit",
        "X-bcc" : "",
        "To" : "[email protected]",
        "X-Origin" : "Allen-P",
        "X-FileName" : "pallen (Non-Privileged).pst",
        "X-From" : "Phillip K Allen",
        "Date" : "Mon, 14 May 2001 16:39:00 -0700 (PDT)",
        "X-To" : "Tim Belden ",
        "Message-ID" : "<18782981.1075855378110.JavaMail.evans@thyme>",
        "Content-Type" : "text/plain; charset=us-ascii",
        "Mime-Version" : "1.0"
    }
}

This is my .Net code for NEST to connect Elasticsearch -

    public class Message
    {
        public string id { get; set; }
        public string body { get; set; }
        public string mailbox { get; set; }
        public string filename { get; set; }

        [ElasticProperty(Type = FieldType.Nested)]
        public IList<HeadersComponent> headers { get; set; }
    }

    public class HeadersComponent
    {
        public string Cc { get; set; }
        public string Bcc { get; set; }
        public string From { get; set; }
        public string Subject { get; set; }
        public string To { get; set; }
        public string Date { get; set; }        
    }


    Uri node = new Uri("http://localhost:9200");
    ConnectionSettings settings = new ConnectionSettings(node, "MongoIndex");
    ElasticClient client = new ElasticClient(settings);

    ISearchResponse<Message> searchResponse = client.Search<Message>(s => s.Query(q => q.Term(p => p.body, "SearchMe"));

When I run the above code to search for "SearchMe" text in the email 'body', it returns 0 results.

Need help on this. Its driving me nuts :)

Can you add the json body for your REST API query and the URI that you are sending the request to? — ceej
Using GET verb, I am querying the database - localhost:9200/mongoindex/_search?q=body:"SearchMe" — user4894970
So I think your queries are not actually identical as you are not querying a type here but across all tyoes in your index.. Have you tried my suggestion in my answer? I suspect that the request being generated for NEST is localhost:9200/mongoindex/message/_search?q=body:"SearchMe" — ceej
If I have a class "Message" in .Net, how it would know it needs to map it to the _Type "Messages" in ES ? — user4894970

ceej ceej · Accepted Answer · 2015-08-18T12:32:23

I think it might be your Elasticsearch type. Looking at the request you provided in your comment it shows that you are querying across all types within your index.

localhost:9200/mongoindex/_search?q=body:"SearchMe"

However NEST will by default add in a type to the request. This will result in the following request string

localhost:9200/mongoindex/message/_search?q=body:"SearchMe"

However this means your query will not match as you actually want

localhost:9200/mongoindex/messages/_search?q=body:"SearchMe"

since you say that your type is Messages.

ES Details- Index: MongoIndex Type: Messages

However, in your query you leave the inference of the type to NEST. By default this will produce a type of Message. There are a number of options to fix this but to confirm this you could try explicitly setting the type to use.

var searchResponse = client.Search<Message>(s => s
    .Type("Messages")
    .Query(q => q
        .Term(p => p.body, "SearchMe")
    )
);

You can see more options for how to influence the way NEST infers types, index names, property names etc. in the documentation.

MongoDB-MongoRiver-ElasticSearch-NEST

Single document format for a message in the collection -

1 Answers