Elasticsearch 1.5.2 deployment issue

Question

I have ES 1.5.2 cluster with the following specs:

3 nodes with RAM: 32GB, CPU cores: 8 each
282 total indices
2,564 total shards
799,505,935 total docs
767.84GB total data
ES_HEAP_SIZE=16g

The problem is when I am using Kibana to query some thing (very simple queries), if it a single query it`s working fine, but if I continue to query some more - elastic is getting so slow and eventually stuck because the JVM heap usage (from Marvel) is getting to 87-95%. It happens also when I trying to load some Kibana dashboard and the only solution for this situation is to restart the service on all the nodes.

(This is also happens on ES 2.2.0 , 1 node, with Kibana 4)

What is wrong, what am I missing? Am I suppose to query less?

EDIT:

I had to mention that I have a lot of empty indices (0 documents) but the shards are counted. This is this way because I set ttl on the documents for 4w, and the empty indices will be deleted with curator.

Also we have not disabled doc_values in 1.5.2 nor 2.2.0 clusters. The accurate specs are as following (1.5.2):

3 nodes with RAM: 32GB, CPU cores: 8 each
282 total indices = 227 empty + 31 marvel + 1 kibana + 23 data
2,564 total shards = (1135 empty + 31 marvel + 1 kibana + 115 data)* 1 replica
799,505,935 total docs
767.84GB total data
ES_HEAP_SIZE=16g

curl _cat/fielddata?v result:

1.5.2:

 total os.cpu.usage primaries.indexing.index_total total.fielddata.memory_size_in_bytes jvm.mem.heap_used_percent jvm.gc.collectors.young.collection_time_in_millis primaries.docs.count device.imei fs.total.available_in_bytes os.load_average.1m index.raw @timestamp node.ip_port.raw fs.total.disk_io_op node.name jvm.mem.heap_used_in_bytes jvm.gc.collectors.old.collection_time_in_millis total.merges.total_size_in_bytes jvm.gc.collectors.young.collection_count jvm.gc.collectors.old.collection_count total.search.query_total 
 2.1gb        1.2mb                          3.5mb                                3.4mb                     1.1mb                                                0b                3.5mb       2.1gb                       1.9mb              1.8mb     3.6mb      3.6mb            1.7mb               1.9mb     1.7mb                      1.6mb                                           1.5mb                            3.5mb                                    1.5mb                                  1.5mb                    3.2mb 
 1.9gb        1.2mb                          3.4mb                                3.3mb                     1.1mb                                             1.5mb                3.5mb       1.9gb                       1.9mb              1.8mb     3.5mb      3.6mb            1.7mb               1.9mb     1.7mb                      1.5mb                                           1.5mb                            3.4mb                                       0b                                  1.5mb                    3.2mb 
   2gb           0b                             0b                                   0b                        0b                                                0b                   0b         2gb                          0b                 0b        0b         0b               0b                  0b        0b                         0b                                              0b                               0b                                       0b                                     0b                       0b

2.2.0:

  total index_stats.index node.id node_stats.node_id buildNum endTime location.timestamp userActivity.time startTime   time shard.state shard.node indoorOutdoor.time shard.index dataThroughput.downloadSpeed 
176.2mb                0b      0b                 0b     232b 213.5kb            518.8kb           479.7kb    45.5mb 80.1mb       1.4kb       920b            348.7kb       2.5kb                       49.1mb

Ok, those are a lot of shards and I'm almost certain that, given the right query with the right time span, you will fill the memory with fielddata. After your reach high heap usage, run GET /_nodes/stats and provide the output somewhere in gist. Provide the link to that output. Do the same in ES 2.x. — Andrei Stefan
@Rada, if the fielddata usage is indeed the problem (and it probably is) one solution would be to add the following settings in your elasticsearch.yml config file: index.fielddata.cache: node and indices.fielddata.cache.size: 30%. 30% is what worked for me with a similarly loaded 1.5 ES cluster, but your mileage may vary. What will happen is that the heap won't be exhausted by fielddata, which will never exceed 30% of the heap. Of course, you'll get plenty of evictions (hence slower queries) but at least you won't need to restart so often. You can also add more nodes to share the load. — Val
Please, delete the empty indices. The major usage of your heap is for fielddata - around 9.5GB for each node, 1.2GB for filter cache and around 1.7GB for segments files' metadata. It's strange that for 2.x you have that amount of fielddata still. ES 2.x is using doc_values by default where possible. The only field type where is not possible is analyzed strings. Also, the gist you provided I'm almost sure is not from the 2.2.0 cluster, it must be the 1.5 cluster. — Andrei Stefan
No, the main problem is the fielddata usage in 1.5.2 cluster. I saw the snippet from the template you provided, but is that template applied to your indices correctly, meaning do you really have string fields set as not_analyzed? Also, it is not enough to make the strings not_analyzed in 1.5.2, you also need to enable doc_values: elastic.co/guide/en/elasticsearch/guide/1.x/… — Andrei Stefan
Until the doc_values will be predominant in your indices in the 1.5 cluster, what @Val suggested is the only option: limit the fielddata cache size or add more nodes to your cluster. Or manually clear the fielddata cache ;-) from time to time. — Andrei Stefan

Andrei Stefan Andrei Stefan · Accepted Answer · 2016-04-13T05:25:39

delete the empty indices
for the 1.5 cluster the major usage of your heap is for fielddata - around 9.5GB for each node, 1.2GB for filter cache and around 1.7GB for segments files' metadata
- even if you have that snippet in your template to make the strings as not_analyzed, in 1.5 this doesn't automatically mean ES will use doc_values, you need to specifically enable them.
- if you enable doc_values now in 1.5.x cluster, the change will be effective with the new indices. For the old indices you need to reindex the data. Or if you have time-based indices (created daily, weekly etc) you just need to wait for the new indices to be created and the old ones to be deleted.
- until the doc_values will be predominant in your indices in the 1.5 cluster, what @Val suggested in the comments is the only option: limit the fielddata cache size or add more nodes to your cluster (and implicitly more memory) or increase the RAM on your nodes. Or manually clear the fielddata cache ;-) from time to time.
not related to the memory issue entirely, but try to avoid using ttl. If you don't need some data anymore, simply delete the index, don't rely on ttl, it is much more costly than simply deleting the index. The use of ttl creates can potentially cause issues at search time and affect the overall performance of a cluster, as it deletes documents from indices, which means a lot of updates and a lot of merging to those indices. Since you probably have time-based indices (which means data from yesterday doesn't really change) using ttl brings unnecessary operations on data that should otherwise be static (and which can potentially be optimized).

Elasticsearch 1.5.2 deployment issue

2 Answers