Can’t add node to a single node cluster in elasticsearch [ master_not_discovered_exception ]

Question

I am currently running a single node cluster with nginx proxy for security reason. I want to add a new node to my cluster. So i install elasticsearch and kibana in my new node and nginx for proxy forward. for port forwarding i use 8000 instead of 9200 . But when i modify the new node unicast variable to my master node. And after restart both the nodes. I got error when i hit the following url

ip:8000/_cluster/health

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

I can see elasticsearch is running when i hit ip:8000

Older Node Configuration:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: <Cluster Name>
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: elasticsearch-24-384-node-1
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /mnt/elastic_data/data
#
# Path to log files:
#
path.logs: /mnt/elastic_data/logs
#
# ----------------------------------- Memory -----------------------------------
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: ["127.0.0.1", <new-node-ip>]
#
# Set a custom port for HTTP:
#
# http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["127.0.0.1", "<new-node-ip>"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true

New node configuration:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: <Cluster Name>
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: elasticsearch-24-384-node-2
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: ["127.0.0.1", <old-node-ip>]
#
# Set a custom port for HTTP:
#
# http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["127.0.0.1", "<Older-node-ip>:9300"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various ----------------------------------

So can any one tell what am i doing wrong ??

Answer 1

There are two port you shoud be able two see from each node:
- 9200 is the default port for REST
- 9300 is the default port for nodes communication

So can each node see anothers node 9300 port?

Answer 2

I gave network.host as current node ip, So that it will publish this ip to connect other nodes.

Node - 1

cluster.name: <Cluster Name>
node.name: <Node Name>
network.host: <node1-ip>
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "<node2-ip>:9300"]
discovery.zen.minimum_master_nodes: 1

Node - 2

cluster.name: <Cluster Name>
node.name: <Node Name - 2>
network.host: <node2-ip>
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "<node1-ip>:9300"]
discovery.zen.minimum_master_nodes: 1

Above configuration will work. But here is the tricky part, if you are try to secure elasticsearch API using Nginx. The transport module not going to allow you. After you modify the network.host from local to node ip. All the APIs can be accessed without basic authentication.

I changed network.host to network.publish_host so that node ip only published for node connections not for API. But i can't connect the nodes. Anyone have idea about it. Thanks in Advance !