Steps to make an existing JNDI HornetQ service as HA?

178
votes

TL;DR

What are the steps to configure a HA-JNDI service with a HornetQ setup? I believe the documentation is a little scattered. I have read through the docs here but doesn't seem to illustrate in detail.

Longer version:

So we have a HornetQ JMS setup along with JNDI. We have say 5 servers, that run the HornetQ JMS master instance with JNDI service on each. On each of these 5 servers, we also have a slave running for some other HornetQ master.

To illustrate:

Server A - HornetQa_master, JNDI, HornetQb_slave
Server B - HornetQb_master, JNDI, HornetQc_slave
Server C - HornetQc_master, JNDI, HornetQd_slave
Server D - HornetQd_master, JNDI, HornetQe_slave
Server E - HornetQe_master, JNDI, HornetQa_slave

Each of these HornetQ servers serve as middleware for our various backend needs, so that means 5 servers, 5 HornetQ master instances, 5 HornetQ slave instances and 5 JNDI servers. The problem, however, with this setup is that if a server host (not just the process, the host itself), say A goes down, ideally the service should fallback to the HornetQ running on server E which hosts A's HornetQ slave. However, to resume as HornetQ master, the HornetQa_slave needs to talk to the JNDI process running on server A (I presume to replicate messages). Since the host A is itself down, the HornetQa_slave running on E has no way to talk to the JNDI on A, and thus, cannot resume as the master process.

Had the JNDI service been highly available, the slave HornetQ process could resume as master as expected. Could anyone kindly point to the docs or illustrate in simple steps how we could convert our existing setup to a HA-JNDI? For what it's worth, I have read multiple sources, but it doesn't seem to illustrate in much detail about how to get going with configuring a HA-JNDI. Please let me know if you need more info about our current setup.

1
javajbossjndihornetq
Where are your client(s) running? Are they running on the same AS instances or from another instance/JVM, or both? – jjhavokk
@jjhavokk they would be running on another JVM – gravetii
Could you enable HornetQ in High Availability mode (active - passive replication)? Couple that with server dynamic discovery and you should have reliable fallback. See docs.jboss.org/hornetq/2.4.0.Final/docs/user-manual/html/… and docs.jboss.org/hornetq/2.4.0.Final/docs/user-manual/html/… – diginoise
What version of jboss are you running? – eis
I see this is really old, but I'm wondering if you found the answer. By now you probably know that the HA requires <forward-when-no-consumers>true</forward-when-no-consumers> to propagate messages, but that the failback to master doesn't work. I've had the same config in weblogic and websphere where the failback works, but does not with jboss. Is there something to set to allow the master to sync and update missed messages so that a proper failback works? – user1442498

1 Answers

1
votes

With the architecture described it seems difficult to me, because indeed you need to reconfigure the slave as master and then you will have a certain outage.

HornetQ HA is provided via a live-backup pair and load-balancing is provided via a cluster.

If you want both HA and load-balancing then you will need 2 live-backup pairs clustered together.

Source: https://developer.jboss.org/thread/254232

You can reference the master not by the hostname but using a virtual IP address, so that in case the master is down, you can reconfigure one of the slaves as master, and start the virtual ip so you don't have to reconfigure the rest of the slaves. (To keep HA even when the master is down, you want to have 2 slaves, so that you can restart one of them as master and still one will be running).

Another way to achieve the same result is with a DNS hostname specific for the master that you can reconfigure to point to a different IP if one host is down. Since DNS is cached, this entries should better be in the 'hosts' file.

If 3 hosts per HA-domain is too much hardware, you can accomplish this easier with virtual servers without the need to purchase more hardware.