Elasticsearch primary shard lost - how to recover?

Question

I'm running with 3 nodes cluster on AWS EC2, one of my nodes crashed and after reboot I see 2900 unassigned shards and cluster state RED.

I configured indices to have 5 shards with 1 replica - and I don't understand why after rebooting the shards are not recovered from the replicas.

I tried to manually migrate shards with elasticsearch reroute API https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html But got errors:

can't cancel 2, failed to find it on node {infra-elasticsearch-1}

can't move 2, failed to find it on node {infra-elasticsearch-1}

[allocate_replica] trying to allocate a replica shard [filebeat-demo00-2018.07.21][2], while corresponding primary shard is still unassigned"}],"type":"illegal_argument_exception","reason":"[allocate_replica] trying to allocate a replica shard [filebeat-demo00-2018.07.21][2], while corresponding primary shard is still unassigned

It's look like the some primary shard was lost (don't exists on disk) and I don't know how to the state back to GREEN.

thanks

Primary shards won't just get deleted if your node crashed. When a node reboots it is common to see unassigned shards but they will be allocated quickly if the node is healthy. Please use the Cat Allocation API (elastic.co/guide/en/elasticsearch/reference/current/…) and Cluster Allocation Explain API (elastic.co/guide/en/elasticsearch/reference/current/…) to get an understanding of why your shards are unassigned. If primary shards are lost, try restoring them from a backup. — ben5556

Srikanta Srikanta · Accepted Answer · 2018-11-29T15:53:34

Make sure the shard allocation is enabled in the active nodes by using the below API request

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

Also you can check if the replica exists for the indexes whose primary shard has been lost by looking at the Indices information of the Monitoring app on Kibana.

To check the undergoing recovery process use the below API

GET /_recovery

Elasticsearch primary shard lost - how to recover?

1 Answers