term frequency of documents with Nest Elasticsearch

Question

I am new in elasticsearch and want to get the top N term frequency of the "content" field of a specific document using Nest elasticsearch. I've searched a lot to find a proper answer that works for me, but I just got that I should use Terms vector and not Term Facet since it counts the terms in the whole set of documents. I know that I should do some settings for Term Vector like below;

[ElasticProperty(Type = Nest.FieldType.attachment, TermVector =Nest.TermVectorOption.with_positions_offsets, Store = true)]
    public Attachment File { get; set; }

I searched for getting the term frequency of a specific document using Nest Elasticsearch a lot but all I found were about Lucene and Solr. I need an example in Nest elasticsearch. I appreciate your help.

One more question; Actually the solution(suggested by Rob) works well when I want to get the Term frequency of a string like the title of my documents. But when I change the target Field to the Content of the documents, I gain no results back! in order to be able to search the content of documents, I followed the answer in this link: ElasticSearch & attachment type (NEST C#) and it works fine and I can search a term through the Content of the document but for getting the TF it does not work; Below is the code for it;

var searchResults = client.TermVector<Document>(t =>t.Id(ID).TermStatistics().Fields(f => f.File));

Does anyone have a solution for it?

Rob Rob · Accepted Answer · 2015-06-26T10:33:10

You can do this by client.TermVector(..). Here is a simple example:

Document class:

public class MyDocument
{
    public int Id { get; set; } 
    [ElasticProperty(TermVector = TermVectorOption.WithPositionsOffsets)]
    public string Description { get; set; }
    [ElasticProperty(Type = FieldType.Attachment, TermVector =TermVectorOption.WithPositionsOffsetsPayloads, Store = true, Index = FieldIndexOption.Analyzed)]
    public Attachment File { get; set; }
}

Index some test data:

var indicesOperationResponse = client.CreateIndex(indexName, c => c
    .AddMapping<MyDocument>(m => m.MapFromAttributes()));

var myDocument = new MyDocument {Id = 1, Description = "test cat test"};
client.Index(myDocument);
client.Index(new MyDocument {Id = 2, Description = "river"});
client.Index(new MyDocument {Id = 3, Description = "test"});
client.Index(new MyDocument {Id = 4, Description = "river"});

client.Refresh();

Retrieve term statistics through NEST:

var termVectorResponse = client.TermVector<MyDocument>(t => t
    .Document(myDocument)
    //.Id(1) //you can specify document by id as well
    .TermStatistics()
    .Fields(f => f.Description));

foreach (var item in termVectorResponse.TermVectors)
{
    Console.WriteLine("Field: {0}", item.Key);

    var topTerms = item.Value.Terms.OrderByDescending(x => x.Value.TotalTermFrequency).Take(10);
    foreach (var term in topTerms)
    {
        Console.WriteLine("{0}: {1}", term.Key, term.Value.TermFrequency);
    }
}

Output:

Field: description
cat: 1
test: 2

Hope it helps.

UPDATE

When I checked mapping for index one thing was interesting:

{
    "my_index" : {
        "mappings" : {
            "mydocument" : {
                "properties" : {
                    "file" : {
                        "type" : "attachment",
                        "path" : "full",
                        "fields" : {
                            "file" : {
                                "type" : "string"
                            },
                            "author" : {
                                "type" : "string"
                            },
                            "title" : {
                                "type" : "string"
                            },
                            "name" : {
                                "type" : "string"
                            },
                            "date" : {
                                "type" : "date",
                                "format" : "dateOptionalTime"
                            },
                            "keywords" : {
                                "type" : "string"
                            },
                            "content_type" : {
                                "type" : "string"
                            },
                            "content_length" : {
                                "type" : "integer"
                            },
                            "language" : {
                                "type" : "string"
                            }
                        }
                    },
                    "id" : {
                        "type" : "integer"
                    }
                }
            }
        }
    }
}

There is no information about term vector.

When I have created index through sense:

PUT http://localhost:9200/my_index/mydocument/_mapping
{
  "mydocument": {
    "properties": {
      "file": {
        "type": "attachment",
        "path": "full",
        "fields": {
          "file": {
            "type": "string",
            "term_vector":"with_positions_offsets",
            "store": true
          }
        }
      }
    }
  }
}

I was able to retrieve term statistics.

Hope I'll be back later with working mapping created through NEST.

UPDATE2

Based on Greg's answer try this fluent mapping:

var indicesOperationResponse = client.CreateIndex(indexName, c => c
        .AddMapping<MyDocument>(m => m
            .MapFromAttributes()
            .Properties(ps => ps
                .Attachment(s => s.Name(p => p.File)
                    .FileField(ff => ff.Name(f => f.File).TermVector(TermVectorOption.WithPositionsOffsets)))))
    );

term frequency of documents with Nest Elasticsearch

1 Answers