Create index with multi field mapping syntax with NEST 2.x

Question

I just can't seem to get the syntax correct for multi field mapping in NEST 2.0--if that's the correct terminology. Every example I've found for mapping seems to be <= the 1.x version of NEST. I'm new to Elasticsearch and NEST, and I've been reading their documentation, but the NEST documentation hasn't been completely updated for 2.x.

Basically, I don't need to index or store the entire type. Some fields I need for indexing only, some fields I'll need to index and retrieve, and some I don't need for indexing, just for retrieval.

MyType
{
    // Index this & allow for retrieval.
    int Id { get; set; } 

    // Index this & allow for retrieval.
    // **Also**, in my searching & sorting, I need to sort on this **entire** field, not just individual tokens.
    string CompanyName { get; set; } 

    // Don't index this for searching, but do store for display.
    DateTime CreatedDate { get; set; }

    // Index this for searching BUT NOT for retrieval/displaying.
    string CompanyDescription { get; set; } 

    // Nest this.
    List<MyChildType> Locations { get; set; }
}

MyChildType
{
    // Index this & allow for retrieval.
    string LocationName { get; set; }

    // etc. other properties.
}

I've have been able to index the entire object and child as-is using the following as an example:

client.Index(item, i => i.Index(indexName));

However, the actual object is a lot larger than this, and I really don't need most of it. I've found this, which looks like what I think I want to do, but in an older version: multi field mapping elasticsearch

I think "mapping" is what I'm going for, but like I said, I'm new to Elasticsearch and NEST and I'm trying to learn the terminology.

Be gentle! :) It's my first time to ask a question on SO. Thanks!

Answer 1

As far as I can see, you don't have any complex types that you are trying map. So you can easily use NEST attributes to map your objects.

Check this out:

[Nest.ElasticsearchType]
public class MyType
{
    // Index this & allow for retrieval.
    [Nest.Number(Store=true)]
    int Id { get; set; }

    // Index this & allow for retrieval.
    // **Also**, in my searching & sorting, I need to sort on this **entire** field, not just individual tokens.
    [Nest.String(Store = true, Index=Nest.FieldIndexOption.Analyzed, TermVector=Nest.TermVectorOption.WithPositionsOffsets)]
    string CompanyName { get; set; }

    // Don't index this for searching, but do store for display.
    [Nest.Date(Store=true, Index=Nest.NonStringIndexOption.No)]
    DateTime CreatedDate { get; set; }

    // Index this for searching BUT NOT for retrieval/displaying.
    [Nest.String(Store=false, Index=Nest.FieldIndexOption.Analyzed)]
    string CompanyDescription { get; set; }

    [Nest.Nested(Store=true, IncludeInAll=true)]
    // Nest this.
    List<MyChildType> Locations { get; set; }
}

[Nest.ElasticsearchType]
public class MyChildType
{
    // Index this & allow for retrieval.
    [Nest.String(Store=true, Index = Nest.FieldIndexOption.Analyzed)]
    string LocationName { get; set; }

    // etc. other properties.
}

After this declaration, to create this mapping in elasticsearch you need to make a call similar to:

var mappingResponse = elasticClient.Map<MyType>(m => m.AutoMap());

With AutoMap() call NEST will read your attributes from your POCO and create a mapping request accordingly.

Also see "Attribute Based Mapping" section from here.

Cheers!

Answer 2

In addition to Colin's and Selçuk's answers, you can also fully control the mapping through the fluent (and object initializer syntax) mapping API. Here's an example based on your requirements

void Main()
{
    var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
    var connectionSettings = new ConnectionSettings(pool);

    var client = new ElasticClient(connectionSettings);

    client.Map<MyType>(m => m
        .Index("index-name")
        .AutoMap()
        .Properties(p => p
            .String(s => s
                .Name(n => n.CompanyName)
                .Fields(f => f
                    .String(ss => ss
                        .Name("raw")
                        .NotAnalyzed()
                    )
                )
            )
            .Date(d => d
                .Name(n => n.CreatedDate)
                .Index(NonStringIndexOption.No)         
            )
            .String(s => s
                .Name(n => n.CompanyDescription)
                .Store(false)
            )
            .Nested<MyChildType>(n => n
                .Name(nn => nn.Locations.First())
                .AutoMap()
                .Properties(pp => pp
                    /* properties of MyChildType */
                )
            )
        )
    );
}

public class MyType
{
    // Index this & allow for retrieval.
    public int Id { get; set; }

    // Index this & allow for retrieval.
    // **Also**, in my searching & sorting, I need to sort on this **entire** field, not just individual tokens.
    public string CompanyName { get; set; }

    // Don't index this for searching, but do store for display.
    public DateTime CreatedDate { get; set; }

    // Index this for searching BUT NOT for retrieval/displaying.
    public string CompanyDescription { get; set; }

    // Nest this.
    public List<MyChildType> Locations { get; set; }
}

public class MyChildType
{
    // Index this & allow for retrieval.
    public string LocationName { get; set; }

    // etc. other properties.
}

This produces the mapping

{
  "properties": {
    "id": {
      "type": "integer"
    },
    "companyName": {
      "type": "string",
      "fields": {
        "raw": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    },
    "createdDate": {
      "type": "date",
      "index": "no"
    },
    "companyDescription": {
      "type": "string",
      "store": false
    },
    "locations": {
      "type": "nested",
      "properties": {
        "locationName": {
          "type": "string"
        }
      }
    }
  }
}

Calling .AutoMap() causes NEST to infer the mapping based on the property types and any attributes applied to them. Then .Properties() overrides any of the inferred mappings. For example

• CompanyName is mapped as a multi_field with the field companyName analyzed using the standard analyzer and companyName.raw not analyzed. You can reference the latter in your queries using .Field(f => f.CompanyName.Suffix("raw"))
• Locations is mapped as a nested type (automapping by default would infer this as an object type mapping). You can then define any specific mappings for MyChildType using .Properties() inside of the Nested<MyChildType>() call.

Answer 3

At the time of writing, Nest does not offer a way to map a property in your class to multiple fields in your document mapping using built in attributes. However, it does provide the facilities needed to do anything with your mappings that you could do if you wrote the JSON yourself.

Here's a solution I've put together for my own needs. It shouldn't be hard to use it as the starting point for whatever you need to do.

First, here's an example of the mapping I want to generate

{
   "product":{
      "properties":{
         "name":{
            "type":"string",
            "index":"not_analyzed",
            "fields":{
               "standard":{
                  "type":"string",
                  "analyzer":"standard"
               }
            }
         }
      }
   }
}

The product document would then have the name field, which is indexed but not analyzed, and the name.standard field, which uses the standard analyzer.

The C# class that I generate the mapping from looks like this

[ElasticsearchType]
public class Product
{
    [WantsStandardAnalysisField]
    public string Name { get; set; }
}

Note the WantsStandardAnalysisField attribute. That's a custom attribute with no special properties added. Literally just:

public class WantsStandardAnalysisField : Attribute {}

If I were to use AutoMap as-is, my custom attribute would be ignored and I would get a mapping that has the name field, but not name.standard. Luckily, AutoMap accepts an instance of IPropertyVisitor. A base class called NoopPropertyVisitor implements the interface and does nothing at all, so you can subclass it and override only the methods you care about. When you use a property visitor with AutoMap, it will generate a document mapping for you but give you a chance to modify it before it gets sent to Elastic Search. All we need to do is look for properties marked with our custom attribute and add a field to them.

Here's an example that does that:

public class ProductPropertyVisitor : NoopPropertyVisitor
{
    public override void Visit(IStringProperty type, PropertyInfo propertyInfo, ElasticsearchPropertyAttributeBase attribute)
    {
        base.Visit(type, propertyInfo, attribute);

        var wsaf = propertyInfo.GetCustomAttribute<WantsStandardAnalysisField>();
        if (wsaf != null)
        {
            type.Index = FieldIndexOption.NotAnalyzed;
            type.Fields = new Properties
            {
                {
                    "standard",
                    new StringProperty
                    {
                        Index = FieldIndexOption.Analyzed,
                        Analyzer = "standard"
                    }
                }
            };
        }
    }
}

As you can see, we can do pretty much anything we want with the generated property, including turning off analysis for the main property and adding a new field with its own settings. For fun, you could add a couple properties to the custom attribute allowing you to specify the name of the field you want and the analyzer to use. You could even modify the code to see if the attribute has been added multiple times, letting you add as many fields as you want.

If you were to run this through any method that generates a mapping using AutoMap, such as:

new TypeMappingDescriptor<Product>().AutoMap(new ProductPropertyVisitor())

You'll get the desired multi-field mapping. Now you can customize mappings to your heart's content. Enjoy!

Answer 4

I think you have at least 2 possibilities to solve your problem:

1. On indexing: Create something like a metadata model, which is stored just for retrieving. See the _source field to limit the return to this field.
2. On searching: Specify the fields you want to query: if you don`t want to query the CreatedDate, just don't include it in your search.

In my case I am using both of these approaches to get very fast results :-)