Installed mapper-size plugin and added it to index mapping but no _size field is showing up in indexes

Question

We are trying to figure out which documents in our Elasticsearch (version 7.0.1) index are consuming the most disk space. We found the mapper-size plugin provided by Elastic. We installed the plugin on all Elasticsearch data/master nodes, and restarted the ES service on each one. We also added the _size field to the index pattern mapping. However, the _size field is not showing up. This index is fed by several Filebeat services running on our application servers, and the index rolls over each night.

We tried creating a brand new index that matches the index pattern. The _size field was present in the mapping:

"application_log_test" : {
"mappings" : {
  "_size" : {
    "enabled" : true
  }

After adding a few test documents, however, the _size field did not show up in the queried documents. We verified that all Elasticsearch nodes came up with the plugin loaded:

[2019-09-16T15:10:45,103][INFO ][o.e.p.PluginsService     ] [node-name-1] loaded plugin [mapper-size]

We are expecting any document added to the index to calculate and display a _size metadata field. This field doesn't display in our output.

Val Val · Accepted Answer · 2019-09-18T15:00:52

The _size field is not added to your source document. You can query it, aggregate it, sort on it, but to actually see its value, you need to do it through script fields. Try to run the query below and you'll see:

GET application_log_test/_search
{
  "query": {
    "range": {
      "_size": { 
        "gt": 10
      }
    }
  },
  "aggs": {
    "sizes": {
      "terms": {
        "field": "_size", 
        "size": 10
      }
    }
  },
  "sort": [
    {
      "_size": { 
        "order": "desc"
      }
    }
  ],
  "script_fields": {
    "size": {
      "script": "doc['_size']"  
    }
  }
}

Installed mapper-size plugin and added it to index mapping but no _size field is showing up in indexes

1 Answers