ElasticSearch index mapping and wildcards

Question

I have a C# application which reads and writes to an ElasticSearch database using NEST. I'm storing documents of type Dictionary<string, object>

I'm also setting some mapping on the fly just after the creation of the index. I'm doing it via NEST calls, see below:

        elastic.Map<Dictionary<string, object>>(m => m
            .Index(indexName)
                .Type("DataRecord")
            .AutoMap().AllField(a => a.Enabled(false))
            .Dynamic()
            .DynamicTemplates(dt => dt
                .DynamicTemplate("pv_values_template", t => t
                    .Match("ch_*")
                    .Mapping(m2 => m2
                        .Number(n => n
                            .Store(false)
                            .DocValues())))));

The call accepts an indexName which is of course the name of my index. I'm doing this to tell ElasticSearch to set specific mappings to the Dictionary<string, object> documents (can't be done in advance as the type is object)

Now, in this project I store documents on different indexes like:

acme_company_2017
acme_company_2016
acme_company_2015
and so on...

Each year a new index is created, the mapping is set and storing stuff is just fine, everytime.

Then problem arise on reading back the documents:

when I read back the documents I'm using a special index which is:

acme_company_*

This way I can retrieve all the stuff for all the years without knowing in advance the years, just get all the data stored in the database. See below:

    var dataRecordsArray = elastic.Search<Dictionary<string, object>>(searchDescriptor => searchDescriptor
            .Index("acme_company_*")
            .Type("DataRecord")
            .Query(query =>
                query.Term(dataRecord => dataRecord["GroupId"], groupId) &&
                query.DateRange(dateRangeQueryDescriptor => dateRangeQueryDescriptor
                    .Field(dataRecord => dataRecord["DateTime"])
                    .GreaterThanWithStartOption(startTime.UtcDateTime, includeStartTime)
                    .LessThanWithEndOption(endTime.UtcDateTime, includeEndTime)))
                    .Size(MaximumNumberOfReturnedDocs)
                    .Sort(dataRecord => dataRecord.Ascending(data => data["DateTime"])))
                    .Documents.ToArray();

When I'm doing this, the ElasticSearch engine is complaining that the index I selected lacks the mapping. I know that acme_company_* is not the same string of acme_company_2017 but shouldn't it be the same, mapping-wise?

This is the actual error I get:

No mapping found for [DateTime] in order to sort on

DateTime is a key on the Dictionary<string, object> document. When I read back the document, if I provide the very same index (like for example acme_company_2015) everything works just fine

Please note that looping through the indexes from a given year to the end year (and joining the results together) is not an option as the system has to be scalable. Moreover, the indexes may span a shorter time, like a month or even less

Gianluca Ghettini Gianluca Ghettini · Accepted Answer · 2017-02-03T09:45:21

I'll answer myself here,

it turned out that when using acme_company_* I was matching even another set of indexes I didn't mention (i.e. acme_company_metadata) which don't have the mapping.

Solution is to match only the acme_company_<year> indexes, excluding acme_company_metadata. It can be done tightening the regex to match only the year

ElasticSearch index mapping and wildcards

1 Answers