Data node can’t find master node ,but master can find data node in Elasticsearch

Question

Here is log info as below:

    [2016-07-29 00:05:43,764][INFO ][cluster.service ] [data_node_144] detected_master {master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true}, added {{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true},}, reason: zen-disco-receive(from master [{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true}])
    [2016-07-29 00:05:43,915][INFO ][discovery.zen ] [data_node_144] master_left [{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true}], reason [transport disconnected]
    [2016-07-29 00:05:43,974][WARN ][discovery.zen ] [data_node_144] master left (reason = transport disconnected), current nodes: {{data_node_144}{LkMlU7XeSqC-j0K_o05iUw}{192.168.201.144}{192.168.201.144:9300}{master=false},{data_node_148}{FKdVDjWjQ4eykqkMNRNxUQ}{192.168.201.148}{192.168.201.148:9300}{master=false},{master_node_102}{zQR04axjRxGFhmVxw3N4Pg}{192.168.201.102}{192.168.201.102:9300}{data=false, master=true},{master_node_103}{v83JEFKaQa6gXrG8o5xfBw}{192.168.201.103}{192.168.201.103:9300}{data=false, master=true},{data_node_145}{dF-4DvGlT22v2vI68PrIkQ}{192.168.201.145}{192.168.201.145:9300}{master=false},{data_node_146}{TDPHZWaRRm-lTfM2EM3bPQ}{192.168.201.146}{192.168.201.146:9300}{master=false},{master_node_101}{eqxEaFh8TeqR4VmhKPdT_g}{192.168.201.101}{192.168.201.101:9300}{data=false, master=true},{data_node_147}{-evmqt_nSV-RSeXTZ2w15w}{192.168.201.147}{192.168.201.147:9300}{master=false},}
    [2016-07-29 00:05:43,974][INFO ][cluster.service ] [data_node_144] removed {{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true},}, reason: zen-disco-master_failed ({master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true})
    [2016-07-29 00:05:43,974][WARN ][cluster.service ] [data_node_144] failed to notify ClusterStateListener
    java.lang.IllegalStateException: master not available when registering auto-generated license
    at org.elasticsearch.license.plugin.core.LicensesService.requestTrialLicense(LicensesService.java:749)
    at org.elasticsearch.license.plugin.core.LicensesService.clusterChanged(LicensesService.java:483)
    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

and the configure of data node:

    cluster.name: es_cluster
    node.name: data_node_144
    node.master: false
    node.data: true
    path.data: /data/es/data
    path.logs: /data/es/logs
    network.host: 0.0.0.0
    network.publish_host: 0.0.0.0
    gateway.recover_after_nodes: 2
    discovery.zen.ping_timeout: 60s
    discovery.zen.ping.unicast.hosts: ["192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.1
    03","192.168.201.144","192.168.201.145","192.168.201.146","192.168.201.147","192.168.201.148" ]
    discovery.zen.minimum_master_nodes: 3

configure of master node:

    cluster.name: es_cluster
    node.name: master_node_200
    node.master: true
    node.data: false
    path.data: /data/es/data
    path.logs: /data/es/logs
    network.host: 0.0.0.0
    network.publish_host: 0.0.0.0
    gateway.recover_after_nodes: 2
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping_timeout: 60s
    discovery.zen.ping.unicast.hosts: ["192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.103" ]
    discovery.zen.minimum_master_nodes: 3
    bootstrap.mlockall: true

it's weird that master node can find the data node.
PS. master node : "192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.103",192.168.201.200" and others are data node.ES version:2.3.2.Open JDK version:1.8.0_101.

Did you install the license plugin on the master node as well? — Andrei Stefan

Andrei Stefan Andrei Stefan · Accepted Answer · 2016-07-29T05:20:59

The license plugin needs to be installed on all the nodes in the cluster, including the master eligible nodes.

If the license expires, it still can be used, but there will be some limitations:

watcher
- PUT / GET watch APIs are disabled, DELETE watch API continues to work
- Watches execute and write to the history
- The actions of the watches don't execute
shield
- Cluster health, cluster stats and indices stats operations are blocked
- All data operations (read and write) continue to work

And, also, these listed here.

But, mainly, the ES cluster will work.

Data node can’t find master node ,but master can find data node in Elasticsearch

1 Answers