Elasticsearch Cluster - No known master node, scheduling a retry

Question

I have a server running elasticsearch and kibana. I have added a second node to form a cluster but only want that second node to replicate data from the master node.

Based on limited documentation on how to do this, I am running into issue on second with following error

[DEBUG][action.admin.indices.get ] [Match] no known master node, scheduling a retry

I am unable to determine the best configuration for both servers to achieve this but this is what I have done so far:

Master Node Config:

cluster.name: elasticsearch
node.master: true
path.data: /local00/elasticsearch/
path.work: /local00/el_temp/
network.host: 0.0.0.0
http.port: 9200
script.disable_dynamic: true

Node 2

cluster.name: elasticsearch
node.master: false
node.data: true
index.number_of_shards: 5
index.number_of_replicas: 1
path.data: /local00/elasticsearch/
path.work: /local00/el_temp/
network.host: 0.0.0.0
http.port: 9200
script.disable_dynamic: true

I am assuming I am missing additional config somewhere. Any help will be much appreciated.

Answer 1

Got it working with following changes answered here How to set up ES cluster?:

Node 1:

cluster.name: mycluster
node.name: "node1"
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1.example.com"]

Node 2:

cluster.name: mycluster
node.name: "node2"
node.master: false
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1.example.com"]

Answer 2

If you are trying to connect additional node to already existed ES cluster, make sure that this node also have all the same ES plugins as another nodes. If not - node cant be fully connected (other nodes can show it as connected but http-commands cant be launched on it) to ES cluster with error like:

[2016-07-21 11:56:59,564][DEBUG][action.admin.cluster.health] [dev-marvel1] no known master node, scheduling a retry
[2016-07-21 11:57:05,313][INFO ][rest.suppressed          ] /_cluster/health Params: {pretty=true}
MasterNotDiscoveredException[waited for [30s]]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$4.onTimeout(TransportMasterNodeAction.java:154)
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
        at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:574)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

In my case problem was with license plugin. After removing it everything became fine.

Answer 3

If you are searching for this because you are working on your local machine then the quickest solution is to free up nodes that are hung by individually killing all the processes. I searched for these processes using.

ps -alx | grep elastic
kill -9 {pid}