Elastic Search - Find document with a conflicting field type

1
votes

I'm using Elastic Search 5.6.2 with Kibana and I'm currently facing a problem

My documents are indexed on the field timestamp which is normally an integer, however recently somebody has logged a document with a timestamp that is not an integer, and Kibana complains of conflicting type.

The discover panels display nothing and the following errors pop:

  • Saved "field" parameter is now invalid. Please select a new field.
  • Discover: "field" is a required parameter

How can I look for the document(s) causing these conflicts so that to find the service creating bad logs ?

1
elasticsearchkibana
Can you show the error that Kibana reports to you? – Val
@Val I updated the question – allan.simon
What I would suggest is go to the management tab and delete the index pattern and recreate it. Can you try and tell us what happens next? – Val

1 Answers

1
votes

The field type (either integer or text/keyword) is not defined on per document basis but rather on per index basis (in the mappings). I guess you are manipulating timeseries data, and you probably have un index per day (or per month or ...).

In Kibana Dev Tools:

  1. List the created indices with GET _cat/indices
  2. For each index (logstash-2017.09.28 in my example) do a GET logstash-2017.09.28/_mapping and check the type of the field in @timestamp

The field type is probably different between indices.

You won't be able to change the field type on created indices. Deleting document won't solve you're problem. The only solution is to drop the index or reindex the whole index with a new field type (in a specific mapping).

To avoid this problem on future indices, the solution is to create an index template with a mapping telling that the field @timestamp is of type date or whatever.