ActiveMQ Artemis. Reliable cluster with synchronous replication

0
votes

I want to configure a cluster with the following expected behavior:

  1. Сluster must be HA ( 3 nodes at least).
  2. I have queues in which it is important to maintain processing order. The consumer always reads this queue in a single thread. If he took the message, then we consider our task completed.
  3. I don't need load balancing - it is important for me to maintain the order of messages.
  4. I want to avoid split-brain.
  5. If we have 3 nodes, then if 1 of the nodes fails, the cluster should continue to work.

I tried following configurations:

  1. master + slave + slave with replication.

It works. But does not solve the problem of split brain

  1. master + slave + slave + Pinger

As far as I understand, this does not give a 100% guarantee of detecting network problems. We can also get split-brain.

  1. 3 pairs of live/backup nodes.

This is solved split brain problem but how can we avoid the following situation:

  1. Producer send message to group A in queue (where important to maintain processing order)
  2. Group A crashed ( 1/3 of all nodes 2/6)
  3. The message stored in the journal of group A
  4. Cluster continue to work;
  5. Producer send message to group B in queue (where important to maintain processing order)
  6. Consumer got this message first; We did not support the required message order.

How should I build a cluster to solve these problems?

1
activemq-artemis

1 Answers

2
votes

You can't achieve the behavior you want using replication. You need to use a shared store between the nodes. If you must use 3 nodes then I would recommend master + slave + slave. Otherwise I'd recommend master + slave.

Also, for what it's worth, replication is not synchronous within the broker. It is asynchronous and non-blocking. However, it is still reliable. For example, when a broker is configured for HA with replication and it receives a durable message from a client it will persist that message to disk and send it to the replicated backup concurrently without blocking. However, it will wait for both operations to finish before responding to the client that it has received the message. This allows much greater message throughput than using a synchronous architecture internally although the whole process will appear to be synchronous to external clients.

Also, it's worth noting that work is underway to change how replication works to make it more robust against split brain and to enable a single master + slave pair that is suitable for production use.