One node in Elasticsearch cluster permanently overloaded

Question

I have an Elasticsearch cluster with 11 nodes. Five of these are data nodes and the other ones are client nodes from where I add and retrieve documents.

I am using the standard Elasticsearch configuration. Each index has 5 shards and replicas. In the cluster I have 55 indices and round about 150GB of data.

The cluster is very slow. With the Kopf plugin I can see the stats of each node. There I can see that one single data node (not the master) is permanently overloaded. Heap, disk, cpu are ok, but load is almost every time 100%. I have noticed, that every shard is a primary shard whereas all other data nodes have both primary shards and replicas. When I shutdown that node and then on again, the same problem occurs at another data node.

And I don't know why and how to solve this problem. I thought that the client nodes and the master node distribute the requests evenly? Why is always one data node overloaded?

Farhad Farahi Farhad Farahi · Accepted Answer · 2017-02-28T08:21:53

Try the following settings:

cluster.routing.rebalance.enable:

Enable or disable rebalancing for specific kinds of shards:

    all - (default) Allows shard balancing for all kinds of shards.
    primaries - Allows shard balancing only for primary shards.
    replicas - Allows shard balancing only for replica shards.
    none - No shard balancing of any kind are allowed for any indices.

cluster.routing.allocation.allow_rebalance:

Specify when shard rebalancing is allowed:

    always - Always allow rebalancing.
    indices_primaries_active - Only when all primaries in the cluster are allocated.
    indices_all_active - (default) Only when all shards (primaries and replicas) in the cluster are allocated.

cluster.routing.allocation.cluster_concurrent_rebalance:

Allow to control how many concurrent shard rebalances are allowed cluster wide.

Defaults to 2

Sample curl to apply desired settings:

curl -XPUT <elasticsearchserver>:9200/_cluster/settings -d '{ 
  "transient" : {
    "cluster.routing.rebalance.enable" : "all"
  }
}

You can replace transient with persistent if you want your settings persist across restarts.

One node in Elasticsearch cluster permanently overloaded

1 Answers