Status yellow elasticsearch after adding and removing a node

Question

I have a elasticsearch cluster with 2 data nodes with one replica node (green status). If I add a new node, the status will still be green. If I shutdown elasticsearch on the newly added node, I get a yellow status. How could I make understand to revert back to 2 nodes instead of 3 and get a green status?

Before shutdown:

{
  "cluster_name" : "elastic_poiuytrez",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 65,
  "active_shards" : 130,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

curl -XGET http://localhost:9200/_cluster/settings?pretty=true
{
  "persistent" : { },
  "transient" : { }
}

This is what I get when I shutdown one of the node:

{
  "cluster_name" : "elastic_poiuytrez",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 65,
  "active_shards" : 87,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 43,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

Nodes settings: http://pastebin.com/wpmsRAbK (elastic-node-1 is down)

That's weird. Yellow for a short period of time or yellow forever? — Andrei Stefan
Can you, also, post the output of GET /_nodes/settings? It will be a bigger one. — Andrei Stefan
Hm, no red flags as I see. Anything interesting in the logs, maybe? — Andrei Stefan

Jonesome Reinstate Monica Jonesome Reinstate Monica · Accepted Answer · 2015-11-24T23:41:17

From my experience of cluster status green/yellow/red:

a) Depending on number of replicas: Some replicas have been put on the new node, and have to be recreated or moved to remaining node(s)

b) If ES ran tight on memory, and for other reasons (like removing a node) one or more replicas can go to unassigned. The cluster will be yellow until "unassigned_shards" goes down to 0

Looking at your post, notice:

  "unassigned_shards" : 43,

That will cause the cluster to be in yellow status. It may take some time to get everything shuffled around on the remaining nodes.

Watching "unassigned_shards" can show you "progress toward green"

Status yellow elasticsearch after adding and removing a node

1 Answers