Elasticsearch Unassigned Shard: How to reassign back?

Question

We have 3 data node cluster and all our indices have 1 primary + 2 replica shards on each index. All the indices were running properly with a green status for the cluster until a couple of days ago.

For some reason, only one of the indices now has a yellow status with 1 unassigned index error. When I run _cluster/allocation/explain, it tells me that "the shard cannot be allocated to the same node on which a copy of the shard already exists". If I look at the cluster health on Kibana, I don't see a duplicate shard on that node.

So I am not quite sure what the problem is. Can someone offer any advice on this?

Answer 1

This statement is correct.

the shard cannot be allocated to the same node on which a copy of the shard already exists as it doesn't make sense to assign the replicas(R) on the same node where primary shard(P) of a index is present, as replicas are used to provide the high-availability and fault-tolerance and always assigned to the node where replica's primary shard is not present.

Now in your case, you mentioned you have 3 data node and all your indices also have 3 shards(1 P+ 2R) which makes it ideal to assign 1 shard(be it P or R shard) on each data nodes.

Also you are correct that cluster health on Kibana, I don't see a duplicate shard on that node. and it can't happen that P or R can be on same node.

As you have already used the _cluster/allocation/explain I am assuming that you already know the problematic index, so just follow the advise mentioned in this discuss post, In short, reduce the no of replica for problematic index to just 1 or increase data node.

Answer 2

Try closing the index and open again. It force to redistribute shards, al least on my cluster.