I have 3 nodes of elasticsearch all of them act as master-data node. Due to connectivity issue one node leaves the cluster and promotes iteslf as master.Now i have two cluster first one with two nodes and other with one node. As all the nodes were under load balancer all nodes were receiving request from logstash.What will happen if i restart the single node cluster and try to add it back to the original cluster?
0
votes
1 Answers
0
votes
The problem that you are encountering is called split brain problem.
Here is a description of it
The problem comes in when a node falls down or there's simply a lapse in communication between nodes for some reason. If one of the slave nodes cannot communicate with the master node, it initiates the election of a new master node from those it's still connected with. That new master node then will take over the duties of the previous master node. If the older master node rejoins the cluster or communication is restored, the new master node will demote it to a slave so there's no conflict. For the most part, this process is seamless and "just works."
However, consider a scenario where you have just two nodes: one master and one slave. If communication between the two is disrupted, the slave will be promoted to a master, but once communication is restored, you end up with two master nodes. The original master node thinks the slave dropped and should rejoin as a slave, while the new master thinks the original master dropped and should rejoin as a slave. Your cluster, therefore, is said to have a split brain.
Reference link to it : https://qbox.io/blog/split-brain-problem-elasticsearch
To avoid this problem add this to your yml file on your master nodes : discovery.zen.minimum_master_nodes: 2
The formulae for this is : Prevent the "split brain" by configuring the
majority of nodes (total number of master-eligible nodes / 2 + 1)
