Elasticsearch not allocating shards nor replicas properly

Question

I am running a 2 node elasticsearch cluster and have all my indices configurured with 2 primary shards and 1 replica. At first I thought that every node would store 1 primary shard and 1 replica, although that is not what is happenning.

 curl -XGET http://localhost:9200/_cat/shards
 .kibana                   0 p STARTED       1   3.1kb 10.151.6.98 Eleggua
 .kibana                   0 r UNASSIGNED
 logstash-sflow-2016.10.03 1 p STARTED     738 644.4kb 10.151.6.98 Eleggua
 logstash-sflow-2016.10.03 1 r UNASSIGNED
 logstash-sflow-2016.10.03 0 p STARTED     783 618.4kb 10.151.6.98 Eleggua
 logstash-sflow-2016.10.03 0 r UNASSIGNED
 logstash-ipf-2016.10.03   1 p STARTED    8480   3.9mb 10.151.6.98 Eleggua
 logstash-ipf-2016.10.03   1 r UNASSIGNED
 logstash-ipf-2016.10.03   0 p STARTED    8656   6.3mb 10.151.6.98 Eleggua
 logstash-ipf-2016.10.03   0 r UNASSIGNED
 logstash-raw-2016.10.03   1 p STARTED     254 177.9kb 10.151.6.98 Eleggua
 logstash-raw-2016.10.03   1 r UNASSIGNED
 logstash-raw-2016.10.03   0 p STARTED     274   180kb 10.151.6.98 Eleggua
 logstash-raw-2016.10.03   0 r UNASSIGNED
 logstash-pf-2016.10.03    1 p STARTED    4340   2.9mb 10.151.6.98 Eleggua
 logstash-pf-2016.10.03    1 r UNASSIGNED
 logstash-pf-2016.10.03    0 p STARTED    4234   5.7mb 10.151.6.98 Eleggua
 logstash-pf-2016.10.03    0 r UNASSIGNED

As shown above, every shard is hosted by a single node, and no replica is assigned.

curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "es_gts_seginfo",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 9,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 9,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

What am I doing wrong?

Could you post your cluster settings? Are you seeing anything in the elasticsearch logs? What is the output of curl -XPOST "http://localhost:9200/_cluster/reroute?explain — Alon Torres
Have you tried the steps mentioned here: stackoverflow.com/a/23816954/689625 . Did u put a bound on the number of shards per node? elastic.co/guide/en/elasticsearch/reference/current/… — jay
Can you show the network configuration of your nodes? Do they "see" each other, i.e. have they discovered each other? — Val

bananabr bananabr · Accepted Answer · 2016-10-04T14:51:45

Thanks to you all I was able to figure out the problem. One of my nodes was running 2.4.0 and the other one 2.4.1. This way rerouting was not working properly.

curl -XPOST -d '{ "commands" : [ {
>   "allocate" : {
>        "index" : ".kibana",
>        "shard" : 0,
>        "node" : "proc-gts-elk01",
>        "allow_primary":true
>      }
>   } ] }' http://localhost:9200/_cluster/reroute?pretty
{
  "error" : {
    "root_cause" : [ {
      "type" : "illegal_argument_exception",
      "reason" : "[allocate] allocation of [.kibana][0] on node {proc-gts-elk01}{dhLrHPqTR0y9IkU_kFS5Cw}{10.151.6.19}{10.151.6.19:9300}{max_local_storage_nodes=1, hostname=proc-gts-elk01, data=yes, master=yes} is not allowed, reason: [YES(below shard recovery limit of [2])][YES(node passes include/exclude/require filters)][YES(primary is already active)][YES(enough disk for shard on node, free: [81.4gb])][YES(shard not primary or relocation disabled)][YES(shard is not allocated to same node or host)][YES(allocation disabling is ignored)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(node meets awareness requirements)][YES(allocation disabling is ignored)][NO(target node version [2.4.0] is older than source node version [2.4.1])]"
    } ],
    "type" : "illegal_argument_exception",
    "reason" : "[allocate] allocation of [.kibana][0] on node {proc-gts-elk01}{dhLrHPqTR0y9IkU_kFS5Cw}{10.151.6.19}{10.151.6.19:9300}{max_local_storage_nodes=1, hostname=proc-gts-elk01, data=yes, master=yes} is not allowed, reason: [YES(below shard recovery limit of [2])][YES(node passes include/exclude/require filters)][YES(primary is already active)][YES(enough disk for shard on node, free: [81.4gb])][YES(shard not primary or relocation disabled)][YES(shard is not allocated to same node or host)][YES(allocation disabling is ignored)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(node meets awareness requirements)][YES(allocation disabling is ignored)][NO(target node version [2.4.0] is older than source node version [2.4.1])]"
  },
  "status" : 400
}

Elasticsearch not allocating shards nor replicas properly

1 Answers