Connectivity issue with ActiveMQ Artemis HA backup server

1
votes

I'm having some trouble connecting to the fallback server in an HA cluster. I have just a primary and secondary. I'm using stomp.py and JMS to test.

I'm using virtual box and a simulated local network to experiment with configuration before setting up HA on our cloud servers. I can connect with the primary using the IP like so:

stomp -H 192.168.56.105 -P 61616 -U test -W password

But after shutting down the primary, I'm not able to connect with the secondary. I'm not sure if it's actually activating or not.

stomp -H 192.168.56.106 -P 61616 -U test -W password

This gives "No connection could be made because the target machine actively refused it Could not connect to host 192.168.56.106, port 61616"

Also, I've been using the ha > replicated-failback example, and I've read the documentation for clusters and HA several times trying to figure out what I'm missing.

Here is how I'm trying to confirm UDP connectivity. To me it looks like the acceptors are appearing, but not the cluster on the network.

Primary:192.168.56.105$ sudo nmap -sU -p 61616 192.168.56.106
    
    Starting Nmap 7.60 ( https://nmap.org ) at 2020-06-24 18:16 UTC
    Nmap scan report for 192.168.56.106
    Host is up (-0.15s latency).
    
    PORT      STATE  SERVICE
    61616/udp closed unknown
    MAC Address: 08:00:27:7B:56:3F (Oracle VirtualBox virtual NIC)
    
    Nmap done: 1 IP address (1 host up) scanned in 0.48 seconds

Primary:192.168.56.105$ sudo nmap -sU -Pn -p 9876 231.7.7.7

Starting Nmap 7.60 ( https://nmap.org ) at 2020-06-24 19:25 UTC
Nmap done: 1 IP address (0 hosts up) scanned in 0.45 seconds

Backup:192.168.56.106$ sudo nmap -sU -p 61616 192.168.56.105

Starting Nmap 7.60 ( https://nmap.org ) at 2020-06-24 19:28 UTC
Nmap scan report for 192.168.56.105
Host is up (-0.15s latency).

PORT      STATE  SERVICE
61616/udp closed unknown
MAC Address: 08:00:27:0F:36:BE (Oracle VirtualBox virtual NIC)

Nmap done: 1 IP address (1 host up) scanned in 0.49 seconds

Backup:192.168.56.106$ sudo nmap -sU -Pn -p 9876 231.7.7.7

Starting Nmap 7.60 ( https://nmap.org ) at 2020-06-24 19:29 UTC
Nmap done: 1 IP address (0 hosts up) scanned in 0.46 seconds

Any ideas? My configuration is below.

Primary (192.168.56.105) broker.xml:

<configuration xmlns="urn:activemq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xi="http://www.w3.org/2001/XInclude"
               xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">

    <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:activemq:core ">
        <name>0.0.0.0</name>
        <persistence-enabled>true</persistence-enabled>
        <journal-type>NIO</journal-type>
        <paging-directory>data/paging</paging-directory>
        <bindings-directory>data/bindings</bindings-directory>
        <journal-directory>data/journal</journal-directory>
        <large-messages-directory>data/large-messages</large-messages-directory>
        <journal-datasync>true</journal-datasync>
        <journal-min-files>2</journal-min-files>
        <journal-pool-files>10</journal-pool-files>
        <journal-device-block-size>4096</journal-device-block-size>
        <journal-file-size>10M</journal-file-size>
        <journal-buffer-timeout>2884000</journal-buffer-timeout>
        <journal-max-io>1</journal-max-io>
        <disk-scan-period>5000</disk-scan-period>
        <max-disk-usage>90</max-disk-usage>
        <critical-analyzer>true</critical-analyzer>
        <critical-analyzer-timeout>120000</critical-analyzer-timeout>
        <critical-analyzer-check-period>60000</critical-analyzer-check-period>
        <critical-analyzer-policy>HALT</critical-analyzer-policy>      
        <page-sync-timeout>2884000</page-sync-timeout>
        
        <acceptors>
            <!-- Acceptor for every supported protocol -->
            <acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
        
    <!-- AMQP Acceptor.  Listens on default AMQP port for AMQP traffic.-->
         <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>

         <!-- STOMP Acceptor. -->
         <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>

         <!-- HornetQ Compatibility Acceptor.  Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
         <acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>

         <!-- MQTT Acceptor -->
         <acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>

        </acceptors>

        <connectors>
            <connector name="artemis">tcp://192.168.56.105:61616</connector>
        </connectors>

        <broadcast-groups>
            <broadcast-group name="broadcast-group-1">
                <group-address>231.7.7.7</group-address> 
                <group-port>9876</group-port>
                <connector-ref>artemis</connector-ref>
            </broadcast-group>
        </broadcast-groups>

        <discovery-groups>
            <discovery-group name="discovery-group-1">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
            </discovery-group>
        </discovery-groups>

        <cluster-user>cluster.user</cluster-user>
        <cluster-password>password</cluster-password>

        <ha-policy>
            <replication>
                <master>
                    <check-for-live-server>true</check-for-live-server>
                </master>
            </replication>
        </ha-policy>

        <cluster-connections>
            <cluster-connection name="cluster-1">
                <connector-ref>artemis</connector-ref>
                <discovery-group-ref discovery-group-name="discovery-group-1"/>
            </cluster-connection> 
        </cluster-connections>

        <security-settings>
            <security-setting match="#">
                <permission type="createNonDurableQueue" roles="amq"/>
                <permission type="deleteNonDurableQueue" roles="amq"/>
                <permission type="createDurableQueue" roles="amq"/>
                <permission type="deleteDurableQueue" roles="amq"/>
                <permission type="createAddress" roles="amq"/>
                <permission type="deleteAddress" roles="amq"/>
                <permission type="consume" roles="amq"/>
                <permission type="browse" roles="amq"/>
                <permission type="send" roles="amq"/>
                <!-- we need this otherwise ./artemis data imp wouldn't work -->
                <permission type="manage" roles="amq"/>
            </security-setting>
        </security-settings>

        <address-settings>
            <!-- if you define auto-create on certain queues, management has to be auto-create -->
            <address-setting match="activemq.management#">
                <dead-letter-address>DLQ</dead-letter-address>
                <expiry-address>ExpiryQueue</expiry-address>
                <redelivery-delay>0</redelivery-delay>
                <!-- with -1 only the global-max-size is in use for limiting -->
                <max-size-bytes>-1</max-size-bytes>
                <message-counter-history-day-limit>10</message-counter-history-day-limit>
                <address-full-policy>PAGE</address-full-policy>
                <auto-create-queues>true</auto-create-queues>
                <auto-create-addresses>true</auto-create-addresses>
                <auto-create-jms-queues>true</auto-create-jms-queues>
                <auto-create-jms-topics>true</auto-create-jms-topics>
            </address-setting>
            <!--default for catch all-->
            <address-setting match="#">
                <dead-letter-address>DLQ</dead-letter-address>
                <expiry-address>ExpiryQueue</expiry-address>
                <redelivery-delay>0</redelivery-delay>
                <!-- with -1 only the global-max-size is in use for limiting -->
                <max-size-bytes>-1</max-size-bytes>
                <message-counter-history-day-limit>10</message-counter-history-day-limit>
                <address-full-policy>PAGE</address-full-policy>
                <auto-create-queues>true</auto-create-queues>
                <auto-create-addresses>true</auto-create-addresses>
                <auto-create-jms-queues>true</auto-create-jms-queues>
                <auto-create-jms-topics>true</auto-create-jms-topics>
            </address-setting>
        </address-settings>
        <addresses>
            <address name="DLQ">
                <anycast>
                    <queue name="DLQ" />
                </anycast>
            </address>
            <address name="ExpiryQueue">
                <anycast>
                    <queue name="ExpiryQueue" />
                </anycast>
            </address>

        </addresses>
    </core>
</configuration>

Backup (192.168.56.106) broker.xml:

<configuration xmlns="urn:activemq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xi="http://www.w3.org/2001/XInclude"
               xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">

   <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:activemq:core ">

      <name>0.0.0.0</name>
      <persistence-enabled>true</persistence-enabled>
      <journal-type>NIO</journal-type>
      <paging-directory>data/paging</paging-directory>
      <bindings-directory>data/bindings</bindings-directory>
      <journal-directory>data/journal</journal-directory>
      <large-messages-directory>data/large-messages</large-messages-directory>
      <journal-datasync>true</journal-datasync>
      <journal-min-files>2</journal-min-files>
      <journal-pool-files>10</journal-pool-files>
      <journal-device-block-size>4096</journal-device-block-size>
      <journal-file-size>10M</journal-file-size>
      <journal-buffer-timeout>2868000</journal-buffer-timeout>
      <journal-max-io>1</journal-max-io>
      <disk-scan-period>5000</disk-scan-period>
      <max-disk-usage>90</max-disk-usage>
      <critical-analyzer>true</critical-analyzer>
      <critical-analyzer-timeout>120000</critical-analyzer-timeout>
      <critical-analyzer-check-period>60000</critical-analyzer-check-period>
      <critical-analyzer-policy>HALT</critical-analyzer-policy>      
      <page-sync-timeout>2868000</page-sync-timeout>

      <acceptors>
         <!-- Acceptor for every supported protocol -->
         <acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
        <!-- AMQP Acceptor.  Listens on default AMQP port for AMQP traffic.-->
         <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>

         <!-- STOMP Acceptor. -->
         <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>

         <!-- HornetQ Compatibility Acceptor.  Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
         <acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>

         <!-- MQTT Acceptor -->
         <acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>

      </acceptors>

      <connectors>
          <connector name="artemis">tcp://192.168.56.106:61616</connector>
      </connectors>

      <security-settings>
         <security-setting match="#">
            <permission type="createNonDurableQueue" roles="amq"/>
            <permission type="deleteNonDurableQueue" roles="amq"/>
            <permission type="createDurableQueue" roles="amq"/>
            <permission type="deleteDurableQueue" roles="amq"/>
            <permission type="createAddress" roles="amq"/>
            <permission type="deleteAddress" roles="amq"/>
            <permission type="consume" roles="amq"/>
            <permission type="browse" roles="amq"/>
            <permission type="send" roles="amq"/>
            <!-- we need this otherwise ./artemis data imp wouldn't work -->
            <permission type="manage" roles="amq"/>
         </security-setting>
      </security-settings>

      <address-settings>
         <!-- if you define auto-create on certain queues, management has to be auto-create -->
         <address-setting match="activemq.management#">
            <dead-letter-address>DLQ</dead-letter-address>
            <expiry-address>ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <!-- with -1 only the global-max-size is in use for limiting -->
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <auto-create-queues>true</auto-create-queues>
            <auto-create-addresses>true</auto-create-addresses>
            <auto-create-jms-queues>true</auto-create-jms-queues>
            <auto-create-jms-topics>true</auto-create-jms-topics>
         </address-setting>
         <!--default for catch all-->
         <address-setting match="#">
            <dead-letter-address>DLQ</dead-letter-address>
            <expiry-address>ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <!-- with -1 only the global-max-size is in use for limiting -->
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <auto-create-queues>true</auto-create-queues>
            <auto-create-addresses>true</auto-create-addresses>
            <auto-create-jms-queues>true</auto-create-jms-queues>
            <auto-create-jms-topics>true</auto-create-jms-topics>
         </address-setting>
      </address-settings>

      <addresses>
         <address name="DLQ">
            <anycast>
               <queue name="DLQ" />
            </anycast>
         </address>
         <address name="ExpiryQueue">
            <anycast>
               <queue name="ExpiryQueue" />
            </anycast>
         </address>
      </addresses>

        <broadcast-groups>
        <broadcast-group name="broadcast-group-1">
            <group-address>231.7.7.7</group-address>
            <group-port>9876</group-port>
            <connector-ref>artemis</connector-ref>
        </broadcast-group>
    </broadcast-groups>

    <discovery-groups>
        <discovery-group name="discovery-group-1">
            <group-address>231.7.7.7</group-address>
            <group-port>9876</group-port>
        </discovery-group>
    </discovery-groups>

    <cluster-user>cluster.user</cluster-user>
    <cluster-password>password</cluster-password>

    <ha-policy>
        <replication>
            <slave>
                <allow-failback>true</allow-failback>
            </slave>
        </replication>
    </ha-policy>

    <cluster-connections>
        <cluster-connection name="cluster-1">
            <connector-ref>artemis</connector-ref>
            <discovery-group-ref discovery-group-name="discovery-group-1"/>
        </cluster-connection>
    </cluster-connections>
   </core>
</configuration>

Primary Server Logs:

2020-06-24 18:42:46,893 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-06-24 18:42:46,952 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-06-24 18:42:57,165 INFO  [org.apache.activemq.artemis.core.server] AMQ221013: Using NIO Journal
2020-06-24 18:42:57,182 INFO  [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-06-24 18:42:57,267 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-06-24 18:42:57,268 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-06-24 18:42:57,273 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-06-24 18:42:57,273 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-06-24 18:42:57,274 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-06-24 18:42:57,276 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-06-24 18:42:57,810 INFO  [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2020-06-24 18:42:57,811 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2020-06-24 18:42:57,812 INFO  [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2020-06-24 18:42:57,812 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2020-06-24 18:42:58,358 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2020-06-24 18:42:58,363 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5445 for protocols [HORNETQ,STOMP]
2020-06-24 18:42:58,373 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5672 for protocols [AMQP]
2020-06-24 18:42:58,387 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1883 for protocols [MQTT]
2020-06-24 18:42:58,408 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61613 for protocols [STOMP]
2020-06-24 18:42:58,409 INFO  [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-06-24 18:42:58,409 INFO  [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.13.0 [0.0.0.0, nodeID=4c8091c1-b230-11ea-99de-080027649461] 
2020-06-24 18:42:59,001 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-06-24 18:42:59,110 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-06-24 18:42:59,757 INFO  [io.hawt.HawtioContextListener] Initialising hawtio services
2020-06-24 18:42:59,842 INFO  [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-06-24 18:42:59,843 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-06-24 18:42:59,887 INFO  [io.hawt.jmx.UploadManager] Using file upload directory: /var/lib/broker-1/tmp/uploads
2020-06-24 18:42:59,916 INFO  [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-06-24 18:42:59,962 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/var/lib/broker-1/etc/jolokia-access.xml]
2020-06-24 18:42:59,999 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-06-24 18:43:00,195 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 192.168.56.105, 10.0.3.15]
2020-06-24 18:43:00,790 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-06-24 18:43:00,790 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-06-24 18:43:00,791 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-06-24 19:05:30,551 INFO  [io.hawt.HawtioContextListener] Destroying hawtio services
2020-06-24 19:05:30,562 INFO  [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
2020-06-24 19:05:30,624 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-06-24 19:05:30,630 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-06-24 19:05:30,672 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.13.0 [4c8091c1-b230-11ea-99de-080027649461] stopped, uptime 22 minutes

Backup Server Logs:

        2020-06-24 18:32:45,363 INFO  [io.hawt.HawtioContextListener] Destroying hawtio services
2020-06-24 18:32:45,380 INFO  [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
2020-06-24 18:32:45,488 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-06-24 18:32:45,499 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-06-24 18:32:45,618 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.13.0 [null] stopped, uptime 5 hours 44 minutes
2020-06-24 18:32:49,520 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-06-24 18:32:49,591 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-06-24 18:32:49,633 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /var/lib/broker-1/data/journal/oldreplica.13
2020-06-24 18:32:49,634 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /var/lib/broker-1/data/journal to /var/lib/broker-1/data/journal/oldreplica.15
2020-06-24 18:32:49,717 INFO  [org.apache.activemq.artemis.core.server] AMQ221013: Using NIO Journal
2020-06-24 18:32:49,856 INFO  [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-06-24 18:32:50,131 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-06-24 18:32:50,132 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-06-24 18:32:50,140 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-06-24 18:32:50,140 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-06-24 18:32:50,141 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-06-24 18:32:50,142 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-06-24 18:32:50,547 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-06-24 18:32:50,697 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-06-24 18:32:51,507 INFO  [io.hawt.HawtioContextListener] Initialising hawtio services
2020-06-24 18:32:51,612 INFO  [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-06-24 18:32:51,616 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-06-24 18:32:51,655 INFO  [io.hawt.jmx.UploadManager] Using file upload directory: /var/lib/broker-1/tmp/uploads
2020-06-24 18:32:51,690 INFO  [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-06-24 18:32:51,726 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/var/lib/broker-1/etc/jolokia-access.xml]
2020-06-24 18:32:51,774 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-06-24 18:32:51,962 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 192.168.56.106, 10.0.3.15]
2020-06-24 18:32:52,630 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-06-24 18:32:52,631 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-06-24 18:32:52,631 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
1
virtualboxhigh-availabilityactivemq-artemis
Have you confirmed that the backup is activating when you stop the live? You can do this by looking at the logs of the backup. – Justin Bertram
I've checked [broker]/log/artemis.log on the secondary. After shutting down the primary, nothing at all prints to this log though. Is this the correct one? – BenW
If nothing prints to the log on the backup when the live goes down then that means the 2 brokers haven't paired correctly. Is each broker running in its own VM? If so, can you confirm that UDP is working between them? Also, can you post the logs from both the live and the backup which include the broker start-up process? – Justin Bertram
I restarted both servers, then shut down the primary. After that I copied that portion of the log to my post. I also put the output from trying to confirm UDP using nmap, although I'm not sure I'm using it correctly. It looks like the acceptors are appearing on the network, but not the discovery and broadcast group -- however the broadcast group and cluster appear in the console. – BenW
Thanks Justin. Good idea. I didn't realize even that we could set this up static. That worked on the first try. – BenW

1 Answers

1
votes

I don't see any evidence in your logs that the HA pair is actually forming. Try using static connectors rather than discovery. You could also try running the brokers in the same VM or simply on your local machine. Once you get it working in a simpler environment then you can move to a more complex one.