Kibana and the _source field, Elastic Version 6.7

Question

I want a simple Pie chart based on my Index. However the fields in the result seem to be embedded within the _source field, which cannot be used in a Terms Aggregation in Kibana.

Sample Result is shown below:

Now if I disable the _source field in the mapping:

I don't get any of the fields:

However, the Kibana Discover page is listing the available fields, which are never returned by the ES results - when _source was enabled.

The Index Mapping is as shown below:

{
  "settings": {
    "analysis": {
      "filter": {
        "filter_stemmer": {
          "type": "stemmer",
          "language": "english"
        }
      },
      "analyzer": {
        "tags_analyzer": {
          "type": "custom",
          "filter": [
            "standard",
            "lowercase",
            "filter_stemmer"
          ],
          "tokenizer": "standard"
        }
      }
    }
  },
  "mappings": {
    "schemav1": {
      "properties": {
      "user_id": {
            "type": "text"
        },

        "technician_query": {
          "analyzer": "tags_analyzer",
          "type": "text"
        },
        "staffer_queries": {
          "analyzer": "tags_analyzer",
          "type": "text"
        },
        "status":{
            "type":"text"
        }
      }
    }
  }
}

The JSON document you send is stored int he _source field, that's normal. Are you sure you properly created your index pattern in Kibana? — Val
The Kibana Discover page is listing the available fields, which are never returned by the ES results - when _source was enabled. Added screenshot above. — user2849678
Ok, and are you sure that your index pattern in Kibana is refreshed? What do you see there? — Val

Val Val · Accepted Answer · 2019-08-16T11:50:24

Ok, the reason is simple, in order for your fields to be used in aggregations, you need to have a keyword version of them. You cannot aggregate text fields.

Transform your mapping to this:

  "mappings": {
    "schemav1": {
      "properties": {
        "user_id": {
            "type": "keyword"
        },

        "technician_query": {
          "analyzer": "tags_analyzer",
          "type": "text",
          "fields": {
            "raw": {
              "type": "keyword"
            }
          }
        },
        "staffer_queries": {
          "analyzer": "tags_analyzer",
          "type": "text",
          "fields": {
            "raw": {
              "type": "keyword"
            }
          }
        },
        "status":{
            "type":"keyword"
        }
      }
    }
  }

So, user_id and status are now keyword and technician_query.raw and staffer_queries.raw are also `keyword fields, which you can use in terms aggregations, hence in Pie charts as well.

Kibana and the _source field, Elastic Version 6.7

1 Answers