elasticsearch 2 node cluster: proper setup?

Question

I have two nodes, but they are not on the same subnet.

Questions:

1) As I understand it, in a 2 node cluster, both should be set to master?

2) The config below is the right thing to do to let the nodes find each other? (Since nodes not on same subnet)

3) Client apps can attach to either node, and do reads and writes?

4) Is the proposed config below correct? (Can I specify "node.master: true" in both configs? Will this make the discovery happen?)

Proposed config:

Node 1:

    cluster.name: mycluster
    node.name: "node1"
    node.master: true
    node.data: true
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["192.168.100.103"]  # IP of node2

Node 2:

    cluster.name: mycluster
    node.name: "node2"
    node.master: true
    node.data: true
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["192.168.101.103"]   #IP of node1

2018 UPDATE:

We only use 3 and 5 node clusters now.

Answer 1

Q1: The ideal number of master nodes to prevent a split brain situation is to have (N/2) + 1 masters, so in your situation, since N=2 the number of masters is 2 as well.

Q2: The configs are correct, though you don't need to specify node.master: true and node.data: true as both are true by default.

Q3: That's correct

Q4: Also correct.

Finally, the best way to find out is to run your nodes with those configs and see how it behaves.

1. You start node1, check the logs and see that node1 is master (since it's the only node)
2. then you start node2, check the logs and see that node2 joins the cluster
3. then you bring node1 down, check the logs and verify node2 becomes the master
4. then you bring node1 up again and verify it joins the cluster
5. then you bring node2 down and verify node1 becomes the master again.
6. etc...

Answer 2

You should be careful with two nodes.

For example, the default setting for discovery.zen.minimum_master_nodes is 1. That means if the network is disconnected, each of your master nodes( since you have set both can be master), will check this setting and see that as it can see itself ( default minimum master nodes), it can form a cluster. The other node also acts like this and you are "split-brained". With two nodes you should set this to 2, instead of default value 1.

Now assume you have set it to 2. Then in a network disconnect, both master eligible nodes will cease functioning since they will need to see at least one more master eligible node to begin master election process. Your cluster stops working till connection is assumed.

For two node case , I think only one node should be set as master eligible, at least to let it work in a network problem.

Please check [1] for a similar discussion.

UPDATE: @Eitanmg shared the official documentation [2] for the exact same discussion, you should probably read that.

[1] https://blog.trifork.com/2013/10/24/how-to-avoid-the-split-brain-problem-in-elasticsearch/

[2] https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html#high-availability-cluster-design-two-nodes