ElasticSearch: Unassigned Shards, how to fix?

180

votes

I have an ES cluster with 4 nodes:

number_of_replicas: 1
search01 - master: false, data: false
search02 - master: true, data: true
search03 - master: false, data: true
search04 - master: false, data: true

I had to restart search03, and when it came back, it rejoined the cluster no problem, but left 7 unassigned shards laying about.

{
  "cluster_name" : "tweedle",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 15,
  "active_shards" : 23,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 7
}

Now my cluster is in yellow state. What is the best way to resolve this issue?

Delete (cancel) the shards?
Move the shards to another node?
Allocate the shards to the node?
Update 'number_of_replicas' to 2?
Something else entirely?

Interestingly, when a new index was added, that node started working on it and played nice with the rest of the cluster, it just left the unassigned shards laying about.

Follow on question: am I doing something wrong to cause this to happen in the first place? I don't have much confidence in a cluster that behaves this way when a node is restarted.

NOTE: If you're running a single node cluster for some reason, you might simply need to do the following:

curl -XPUT 'localhost:9200/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}'

elasticsearchshardingmaster

123

votes

By default, Elasticsearch will re-assign shards to nodes dynamically. However, if you've disabled shard allocation (perhaps you did a rolling restart and forgot to re-enable it), you can re-enable shard allocation.

# v0.90.x and earlier
curl -XPUT 'localhost:9200/_settings' -d '{
    "index.routing.allocation.disable_allocation": false
}'

# v1.0+
curl -XPUT 'localhost:9200/_cluster/settings' -d '{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}'

Elasticsearch will then reassign shards as normal. This can be slow, consider raising indices.recovery.max_bytes_per_sec and cluster.routing.allocation.node_concurrent_recoveries to speed it up.

If you're still seeing issues, something else is probably wrong, so look in your Elasticsearch logs for errors. If you see EsRejectedExecutionException your thread pools may be too small.

Finally, you can explicitly reassign a shard to a node with the reroute API.

# Suppose shard 4 of index "my-index" is unassigned, so you want to
# assign it to node search03:
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
    "commands": [{
        "allocate": {
            "index": "my-index",
            "shard": 4,
            "node": "search03",
            "allow_primary": 1
        }
    }]
}'

59

votes

OK, I've solved this with some help from ES support. Issue the following command to the API on all nodes (or the nodes you believe to be the cause of the problem):

curl -XPUT 'localhost:9200/<index>/_settings' \
    -d '{"index.routing.allocation.disable_allocation": false}'

where <index> is the index you believe to be the culprit. If you have no idea, just run this on all nodes:

curl -XPUT 'localhost:9200/_settings' \
    -d '{"index.routing.allocation.disable_allocation": false}'

I also added this line to my yaml config and since then, any restarts of the server/service have been problem free. The shards re-allocated back immediately.

FWIW, to answer an oft sought after question, set MAX_HEAP_SIZE to 30G unless your machine has less than 60G RAM, in which case set it to half the available memory.

References

Shard Allocation Awareness

41

votes

This little bash script will brute force reassign, you may lose data.

NODE="YOUR NODE NAME"
IFS=$'\n'
for line in $(curl -s 'localhost:9200/_cat/shards' | fgrep UNASSIGNED); do
  INDEX=$(echo $line | (awk '{print $1}'))
  SHARD=$(echo $line | (awk '{print $2}'))

  curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
     "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
          }
        }
    ]
  }'
done

17

votes

The only thing that worked for me was changing the number_of_replicas (I had 2 replicas, so I changed it to 1 and then changed back to 2).

First:

PUT /myindex/_settings
{
    "index" : {
        "number_of_replicas" : 1
     }
}

Then:

PUT /myindex/_settings
{
    "index" : {
        "number_of_replicas" : 2
     }
}

(I Already asnwered it in this question)

10

votes

I also encountered similar error. It happened to me because one of my data node was full and due to which shards allocation failed. If unassigned shards are there and your cluster is RED and few indices also RED, in that case I have followed below steps and these worked like a champ.
in kibana dev tool-

GET _cluster/allocation/explain

If any unassigned shards are there then you will get details else will throw ERROR.

simply running below command will solve everything-

POST _cluster/reroute?retry_failed

Thanks to -
https://github.com/elastic/elasticsearch/issues/23199#issuecomment-280272888

9

votes

Elasticsearch automatically allocates shards if the below config is set to all. This config can be set using a rest api as well cluster.routing.allocation.enable: all

If even after application of the below config, es fails to assign the shards automatically, then you have to force assign the shards yourself. ES official link for this

I have written a script to force assign all unassigned shards across cluster.

below array contains list of nodes among which you want to balance the unassigned shards

#!/bin/bash
array=( node1 node2 node3 )
node_counter=0
length=${#array[@]}
IFS=$'\n'
for line in $(curl -s 'http://127.0.0.1:9200/_cat/shards'|  fgrep UNASSIGNED); do
    INDEX=$(echo $line | (awk '{print $1}'))
    SHARD=$(echo $line | (awk '{print $2}'))
    NODE=${array[$node_counter]}
    echo $NODE
    curl -XPOST 'http://127.0.0.1:9200/_cluster/reroute' -d '{
        "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
            }
        }
        ]
    }'
    node_counter=$(((node_counter)%length +1))
done

6

votes

I've stuck today with the same issue of shards allocation. The script that W. Andrew Loe III has proposed in his answer didn't work for me, so I modified it a little and it finally worked:

#!/usr/bin/env bash

# The script performs force relocation of all unassigned shards, 
# of all indices to a specified node (NODE variable)

ES_HOST="<elasticsearch host>"
NODE="<node name>"

curl ${ES_HOST}:9200/_cat/shards > shards
grep "UNASSIGNED" shards > unassigned_shards

while read LINE; do
  IFS=" " read -r -a ARRAY <<< "$LINE"
  INDEX=${ARRAY[0]}
  SHARD=${ARRAY[1]}

  echo "Relocating:"
  echo "Index: ${INDEX}"
  echo "Shard: ${SHARD}"
  echo "To node: ${NODE}"

  curl -s -XPOST "${ES_HOST}:9200/_cluster/reroute" -d "{
    \"commands\": [
       {
         \"allocate\": {
           \"index\": \"${INDEX}\",
           \"shard\": ${SHARD},
           \"node\": \"${NODE}\",
           \"allow_primary\": true
         }
       }
     ]
  }"; echo
  echo "------------------------------"
done <unassigned_shards

rm shards
rm unassigned_shards

exit 0

Now, I'm not kind of a Bash guru, but the script really worked for my case. Note, that you'll need to specify appropriate values for "ES_HOST" and "NODE" variables.

6

votes

In my case, the hard disk space upper bound was reached.

Look at this article: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Basically, I ran:

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.info.update.interval": "1m"
  }
}

So that it will allocate if <90% hard disk space used, and move a shard to another machine in the cluster if >95% hard disk space used; and it checks every 1 minute.

5

votes

In my case, when I create a new index then the default number_of_replicas is set as 1. And the number of nodes in my cluster was only one so there was no extra node to create the replica, so the health was turning to yellow. So when I created the index with settings property and set the number_of_replicas as 0. Then it worked fine. Hope this helps.

PUT /customer
{
    "settings": {
        "number_of_replicas": 0
    }
}

4

votes

Maybe it helps someone, but I had the same issue and it was due to a lack of storage space caused by a log getting way too big.

Hope it helps someone! :)

4

votes

I was having this issue as well, and I found an easy way to resolve it.

Get the index of unassigned shards

$ curl -XGET http://172.16.4.140:9200/_cat/shards

Install curator Tools, and use it to delete index

$ curator --host 172.16.4.140 delete indices --older-than 1 \
       --timestring '%Y.%m.%d' --time-unit days --prefix logstash

NOTE: In my case, the index is logstash of the day 2016-04-21

Then check the shards again, all the unassigned shards go away!

3

votes

I had the same problem but the root cause was a difference in version numbers (1.4.2 on two nodes (with problems) and 1.4.4 on two nodes (ok)). The first and second answers (setting "index.routing.allocation.disable_allocation" to false and setting "cluster.routing.allocation.enable" to "all") did not work.

However, the answer by @Wilfred Hughes (setting "cluster.routing.allocation.enable" to "all" using transient) gave me an error with the following statement:

[NO(target node version [1.4.2] is older than source node version [1.4.4])]

After updating the old nodes to 1.4.4 these nodes started to resnc with the other good nodes.

2

votes

I also meet this situation and finally fixed it.

Firstly, I will describe my situation. I have two nodes in ElasticSearch cluster, they can find each other, but when I created a index with settings "number_of_replicas" : 2, "number_of_shards" : 5, ES show yellow signal and unassigned_shards is 5.

The problem occurs because the value of number_of_replicas, when I set its value with 1, all is fine.

2

votes

For me, this was resolved by running this from the dev console: "POST /_cluster/reroute?retry_failed"

.....

I started by looking at the index list to see which indices were red and then ran

"get /_cat/shards?h=[INDEXNAME],shard,prirep,state,unassigned.reason"

and saw that it had shards stuck in ALLOCATION_FAILED state, so running the retry above caused them to re-try the allocation.

1

votes

In my case an old node with old shares was joining the cluster, so we had to shutdown the old node and delete the indices with unassigned shards.

1

votes

I tried several of the suggestions above and unfortunately none of them worked. We have a "Log" index in our lower environment where apps write their errors. It is a single node cluster. What solved it for me was checking the YML configuration file for the node and seeing that it still had the default setting "gateway.expected_nodes: 2". This was overriding any other settings we had. Whenever we would create an index on this node it would try to spread 3 out of 5 shards to the phantom 2nd node. These would therefore appear as unassigned and they could never be moved to the 1st and only node.

The solution was editing the config, changing the setting "gateway.expected_nodes" to 1, so it would quit looking for its never-to-be-found brother in the cluster, and restarting the Elastic service instance. Also, I had to delete the index, and create a new one. After creating the index, the shards all showed up on the 1st and only node, and none were unassigned.

# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
# gateway.expected_nodes: 2
gateway.expected_nodes: 1

0

votes

Might help, but I had this issue when trying to run ES in embedded mode. Fix was to make sure the Node had local(true) set.

0

votes

Another possible reason for unassigned shards is that your cluster is running more than one version of the Elasticsearch binary.

shard replication from the more recent version to the previous versions will not work

This can be a root cause for unassigned shards.

Elastic Documentation - Rolling Upgrade Process

0

votes

I ran into exactly the same issue. This can be prevented by temporarily setting the shard allocation to false before restarting elasticsearch, but this does not fix the unassigned shards if they are already there.

In my case it was caused by lack of free disk space on the data node. The unassigned shards where still on the data node after the restart but they where not recognized by the master.

Just cleaning 1 of the nodes from the disk got the replication process started for me. This is a rather slow process because all the data has to be copied from 1 data node to the other.

0

votes

I tried to delete unassigned shards or manually assign them to particular data node. It didn't work because unassigned shards kept appearing and health status was "red" over and over. Then I noticed that one of the data nodes stuck in "restart" state. I reduce number of data nodes, killed it. Problem is not reproducible anymore.

0

votes

I had two indices with unassigned shards that didn't seem to be self-healing. I eventually resolved this by temporarily adding an extra data-node^[1]. After the indices became healthy and everything stabilized to green, I removed the extra node and the system was able to rebalance (again) and settle on a healthy state.

It's a good idea to avoid killing multiple data nodes at once (which is how I got into this state). Likely, I had failed to preserve any copies/replicas for at least one of the shards. Luckily, Kubernetes kept the disk storage around, and reused it when I relaunched the data-node.

...Some time has passed...

Well, this time just adding a node didn't seem to be working (after waiting several minutes for something to happen), so I started poking around in the REST API.

GET /_cluster/allocation/explain

This showed my new node with "decision": "YES".

By the way, all of the pre-existing nodes had "decision": "NO" due to "the node is above the low watermark cluster setting". So this was probably a different case than the one I had addressed previously.

Then I made the following simple POST^[2]with no body, which kicked things into gear...

POST /_cluster/reroute

Other notes:

Very helpful: https://datadoghq.com/blog/elasticsearch-unassigned-shards
Something else that may work. Set cluster_concurrent_rebalance to 0, then to null -- as I demonstrate here.

^{^[1] Pretty easy to do in Kubernetes if you have enough headroom: just scale out the stateful set via the dashboard.}

^{^[2] Using the Kibana "Dev Tools" interface, I didn't have to bother with SSH/exec shells.}

0

votes

I just first increased the

"index.number_of_replicas"

by 1 (wait until nodes are synced) then decreased it by 1 afterwards, which effectively removes the unassigned shards and cluster is Green again without the risk of losing any data.

I believe there are better ways but this is easier for me.

Hope this helps.

0

votes

When dealing with corrupted shards you can set the replication factor to 0 and then set it back to the original value. This should clear up most if not all your corrupted shards and relocate the new replicas in the cluster.

Setting indexes with unassigned replicas to use a replication factor of 0:

curl -XGET http://localhost:9200/_cat/shards |\
  grep UNASSIGNED | grep ' r ' |\
  awk '{print $1}' |\
  xargs -I {} curl -XPUT http://localhost:9200/{}/_settings -H "Content-Type: application/json" \
  -d '{ "index":{ "number_of_replicas": 0}}'

Setting them back to 1:

curl -XGET http://localhost:9200/_cat/shards |\
  awk '{print $1}' |\
  xargs -I {} curl -XPUT http://localhost:9200/{}/_settings -H "Content-Type: application/json" \
  -d '{ "index":{ "number_of_replicas": 1}}'

Note: Do not run this if you have different replication factors for different indexes. This would hardcode the replication factor for all indexes to 1.

0

votes

This may be a cause of the disk space as well, In Elasticsearch 7.5.2, by default, if disk usage is above 85% then replica shards are not assigned to any other node.

This can be fixed by setting a different threshold or by disabling it either in the .yml or via Kibana

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.threshold_enabled": "false"
  }
}

0

votes

First use cluster health API to get the current health of cluster, where RED means one or more primary shards missing and Yellow means one of more replica shards are missing.

After this use the cluster allocation explain API to know why a particular shard is missing and elasticsearch is not able to allocate it on data-node.

Once you get the exact root cause, try to address the issue, which often requires, changing few cluster settings(mentioned in @wilfred answer earlier) But in some cases, if its replica shards, and you have another copy of same shard(i.e. another replica) available, you can reduce the replica count using update replica setting and later on again increase it, if you need it.

Apart from above, if your cluster allocation API mention it doesn't have a valid data nodes to allocate a shard, than you need to add a new data nodes, or change the shard allocation awareness settings.

ElasticSearch: Unassigned Shards, how to fix?

25 Answers

References