ActiveMQ Artemis HA setup with SSL enabled

0
votes

I have ActiveMQ Artemis configured as a high available cluster (a master at 192.168.2.111 and two slaves at 192.168.2.112 and 192.168.2.113) with replication. I would like to use SSL (with self-signed certificates) between the brokers themselves as well as between the clients and the brokers.

I generated keyStore and trustore keys and included in broker.xml config on each of the cluster like this:

<connectors>
   <connector name="netty-connector">tcp://192.168.2.111:61616?sslEnabled=true;keyStorePath=broker_ks.p12;keyStorePassword=artemis;trustStorePath=broker_ts.p12;trustStorePassword=artemis</connector>
</connectors>

<acceptors>
   <acceptor name="netty-acceptor">tcp://192.168.2.111:61616?sslEnabled=true;keyStorePath=broker_ks.p12;keyStorePassword=artemis;trustStorePath=broker_ts.p12;trustStorePassword=artemis</acceptor>
</acceptors>

The other part of broker.xml is unchanged.

When I start now master and slave I can reach them through "Management console" but in the log of master I see error:

2020-01-29 16:49:19,395 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) on acceptor "netty-acceptor" during protocol handshake with /192.168.2.113:55118 has occurred.

Can somebody please explain how to fix this error?

This is how I'm generating keystore and truststore on each of the cluster nodes:

    # Create a broker key and cert - import the keypair and cert into the broker keystore
openssl req -newkey rsa:2048 -nodes -keyout broker_keypair.pem -x509 -days 65000 -out broker_cert.pem
openssl pkcs12 -inkey broker_keypair.pem -in broker_cert.pem -export -out broker_ks.p12

# Create a client key and cert - import the keypair and cert into the client keystore
openssl req -newkey rsa:2048 -nodes -keyout client_keypair.pem -x509 -days 65000 -out client_cert.pem
openssl pkcs12 -inkey client_keypair.pem -in client_cert.pem -export -out client_ks.p12

# Create a truststore for the broker, and import the client's certificate. This establishes that the broker "trusts" the client:
keytool -import -alias client -keystore broker_ts.p12 -file client_cert.pem -deststoretype pkcs12

# Create a truststore for the client, and import the broker's certificate. This establishes that the client "trusts" the broker:
keytool -import -alias broker -keystore client_ts.p12 -file broker_cert.pem -deststoretype pkcs12

my broker.xml for master

    <?xml version='1.0'?>

<configuration xmlns="urn:activemq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xi="http://www.w3.org/2001/XInclude"
               xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">

   <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:activemq:core ">

      <name>0.0.0.0</name>

      <persistence-enabled>true</persistence-enabled>

      <journal-type>ASYNCIO</journal-type>

      <paging-directory>data/paging</paging-directory>

      <bindings-directory>data/bindings</bindings-directory>

      <journal-directory>data/journal</journal-directory>

      <large-messages-directory>data/large-messages</large-messages-directory>

      <journal-datasync>true</journal-datasync>

      <journal-min-files>2</journal-min-files>

      <journal-pool-files>10</journal-pool-files>

      <journal-device-block-size>4096</journal-device-block-size>

      <journal-file-size>10M</journal-file-size>

      <journal-buffer-timeout>28000</journal-buffer-timeout>

      <journal-max-io>4096</journal-max-io>

      <!-- how often we are looking for how many bytes are being used on the disk in ms -->
      <disk-scan-period>5000</disk-scan-period>

      <max-disk-usage>100</max-disk-usage>

      <!-- should the broker detect dead locks and other issues -->
      <critical-analyzer>true</critical-analyzer>

      <critical-analyzer-timeout>120000</critical-analyzer-timeout>

      <critical-analyzer-check-period>60000</critical-analyzer-check-period>

      <critical-analyzer-policy>HALT</critical-analyzer-policy>


      <page-sync-timeout>1628000</page-sync-timeout>

            <global-max-size>204Mb</global-max-size>
      <!-- Connectors -->

      <connectors>
         <connector name="netty-connector">tcp://artemis01:61616</connector>
      </connectors>

      <acceptors>
        <acceptor name="netty-ssl-acceptor">tcp://artemis01:61616?sslEnabled=true;needClientAuth=true;keyStorePath=/home/vagrant/ssl/server-side-keystore.jks;keyStorePassword=secureexample;trustStorePath=/home/vagrant/ssl/server-side-truststore.jks;trustStorePassword=secureexample</acceptor>
      </acceptors>

      <cluster-connections>
         <cluster-connection name="my-cluster">
            <address>amq</address>
            <connector-ref>netty-connector</connector-ref>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>3</retry-interval-multiplier>
            <use-duplicate-detection>true</use-duplicate-detection>
            <message-load-balancing>STRICT</message-load-balancing>
            <discovery-group-ref discovery-group-name="my-discovery-group"/>
         </cluster-connection>
      </cluster-connections>

    <broadcast-groups>
       <broadcast-group name="my-broadcast-group">
        <local-bind-address>artemis01</local-bind-address>
        <local-bind-port>9876</local-bind-port>
        <group-address>231.7.7.7</group-address>
        <group-port>9876</group-port>
        <broadcast-period>2000</broadcast-period>
        <connector-ref>netty-connector</connector-ref>
       </broadcast-group>
    </broadcast-groups> 

    <discovery-groups>
       <discovery-group name="my-discovery-group">
        <local-bind-address>artemis01</local-bind-address>
         <local-bind-port>9876</local-bind-port>
          <group-address>231.7.7.7</group-address>
          <group-port>9876</group-port>
          <refresh-timeout>10000</refresh-timeout>
       </discovery-group>
    </discovery-groups>

    <network-check-list>artemis01,artemis02,artemis03</network-check-list>  
    <network-check-period>5000</network-check-period>
    <network-check-timeout>1000</network-check-timeout>
    <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command>
    <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command>

      <!-- Other config -->
      <ha-policy>
        <replication>
          <master>
            <check-for-live-server>true</check-for-live-server>
          </master>
        </replication>
      </ha-policy>

      <security-settings>
         <security-setting match="#">
            <permission type="createNonDurableQueue" roles="amq"/>
            <permission type="deleteNonDurableQueue" roles="amq"/>
            <permission type="createDurableQueue" roles="amq"/>
            <permission type="deleteDurableQueue" roles="amq"/>
            <permission type="createAddress" roles="amq"/>
            <permission type="deleteAddress" roles="amq"/>
            <permission type="consume" roles="amq"/>
            <permission type="browse" roles="amq"/>
            <permission type="send" roles="amq"/>
            <!-- we need this otherwise ./artemis data imp wouldn't work -->
            <permission type="manage" roles="amq"/>
         </security-setting>
      </security-settings>

      <addresses>
         <address name="exampleQueue">
            <anycast>
               <queue name="exampleQueue"/>
            </anycast>
         </address>
         <address name="DLQ">
            <anycast>
               <queue name="DLQ" />
            </anycast>
         </address>
         <address name="ExpiryQueue">
            <anycast>
               <queue name="ExpiryQueue" />
            </anycast>
         </address>
      </addresses>

      <address-settings>
         <!-- if you define auto-create on certain queues, management has to be auto-create -->
         <address-setting match="activemq.management#">
            <dead-letter-address>DLQ</dead-letter-address>
            <expiry-address>ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <!-- with -1 only the global-max-size is in use for limiting -->
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <auto-create-queues>true</auto-create-queues>
            <auto-create-addresses>true</auto-create-addresses>
            <auto-create-jms-queues>true</auto-create-jms-queues>
            <auto-create-jms-topics>true</auto-create-jms-topics>
         </address-setting>
         <!--default for catch all-->
         <address-setting match="#">
            <dead-letter-address>DLQ</dead-letter-address>
            <expiry-address>ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <!-- with -1 only the global-max-size is in use for limiting -->
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <auto-create-queues>true</auto-create-queues>
            <auto-create-addresses>true</auto-create-addresses>
            <auto-create-jms-queues>true</auto-create-jms-queues>
            <auto-create-jms-topics>true</auto-create-jms-topics>
         </address-setting>
         <address-setting match="exampleQueue">            
            <dead-letter-address>DLQ</dead-letter-address>                      
            <redelivery-delay>1000</redelivery-delay>    
            <max-delivery-attempts>3</max-delivery-attempts>
            <max-size-bytes>-1</max-size-bytes>
            <page-size-bytes>1048576</page-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
        </address-setting>
      </address-settings>
   </core>
</configuration>

and broker.xml for slave:

    <?xml version='1.0'?>

<configuration xmlns="urn:activemq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xi="http://www.w3.org/2001/XInclude"
               xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">

   <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:activemq:core ">

      <name>0.0.0.0</name>


      <persistence-enabled>true</persistence-enabled>

      <journal-type>ASYNCIO</journal-type>

      <paging-directory>data/paging</paging-directory>

      <bindings-directory>data/bindings</bindings-directory>

      <journal-directory>data/journal</journal-directory>

      <large-messages-directory>data/large-messages</large-messages-directory>

      <journal-datasync>true</journal-datasync>

      <journal-min-files>2</journal-min-files>

      <journal-pool-files>10</journal-pool-files>

      <journal-device-block-size>4096</journal-device-block-size>

      <journal-file-size>10M</journal-file-size>

      <journal-buffer-timeout>28000</journal-buffer-timeout>

      <journal-max-io>4096</journal-max-io>

      <disk-scan-period>5000</disk-scan-period>

      <max-disk-usage>100</max-disk-usage>

      <!-- should the broker detect dead locks and other issues -->
      <critical-analyzer>true</critical-analyzer>

      <critical-analyzer-timeout>120000</critical-analyzer-timeout>

      <critical-analyzer-check-period>60000</critical-analyzer-check-period>

      <critical-analyzer-policy>HALT</critical-analyzer-policy>


      <page-sync-timeout>1628000</page-sync-timeout>

            <global-max-size>204Mb</global-max-size>
      <!-- Connectors -->

      <connectors>
         <connector name="netty-connector">tcp://artemis02:61616</connector>
      </connectors>

      <acceptors>
        <acceptor name="netty-ssl-acceptor">tcp://artemis02:61616?sslEnabled=true;needClientAuth=true;keyStorePath=/home/vagrant/ssl/server-side-keystore.jks;keyStorePassword=secureexample;trustStorePath=/home/vagrant/ssl/server-side-truststore.jks;trustStorePassword=secureexample</acceptor>
      </acceptors>

      <cluster-connections>
         <cluster-connection name="my-cluster">
            <address>amq</address>
            <connector-ref>netty-connector</connector-ref>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>3</retry-interval-multiplier>
            <use-duplicate-detection>true</use-duplicate-detection>
            <message-load-balancing>STRICT</message-load-balancing>
            <discovery-group-ref discovery-group-name="my-discovery-group"/>
         </cluster-connection>
      </cluster-connections>

    <broadcast-groups>
       <broadcast-group name="my-broadcast-group">
        <local-bind-address>artemis02</local-bind-address>
        <local-bind-port>9876</local-bind-port>
        <group-address>231.7.7.7</group-address>
        <group-port>9876</group-port>
        <broadcast-period>2000</broadcast-period>
        <connector-ref>netty-connector</connector-ref>
       </broadcast-group>
    </broadcast-groups> 

    <discovery-groups>
       <discovery-group name="my-discovery-group">
        <local-bind-address>artemis02</local-bind-address>
         <local-bind-port>9876</local-bind-port>
          <group-address>231.7.7.7</group-address>
          <group-port>9876</group-port>
          <refresh-timeout>10000</refresh-timeout>
       </discovery-group>
    </discovery-groups>

    <network-check-list>artemis01,artemis02,artemis03</network-check-list>  
    <network-check-period>5000</network-check-period>
    <network-check-timeout>1000</network-check-timeout>
    <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command>
    <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command>

      <!-- Other config -->
      <ha-policy>
        <replication>
          <slave>
            <allow-failback>true</allow-failback>
            <failback-delay>5000</failback-delay>
          </slave>
        </replication>
      </ha-policy>

      <security-settings>
         <security-setting match="#">
            <permission type="createNonDurableQueue" roles="amq"/>
            <permission type="deleteNonDurableQueue" roles="amq"/>
            <permission type="createDurableQueue" roles="amq"/>
            <permission type="deleteDurableQueue" roles="amq"/>
            <permission type="createAddress" roles="amq"/>
            <permission type="deleteAddress" roles="amq"/>
            <permission type="consume" roles="amq"/>
            <permission type="browse" roles="amq"/>
            <permission type="send" roles="amq"/>
            <!-- we need this otherwise ./artemis data imp wouldn't work -->
            <permission type="manage" roles="amq"/>
         </security-setting>
      </security-settings>

      <addresses>
         <address name="exampleQueue">
            <anycast>
               <queue name="exampleQueue"/>
            </anycast>
         </address>
         <address name="DLQ">
            <anycast>
               <queue name="DLQ" />
            </anycast>
         </address>
         <address name="ExpiryQueue">
            <anycast>
               <queue name="ExpiryQueue" />
            </anycast>
         </address>
      </addresses>

      <address-settings>
         <!-- if you define auto-create on certain queues, management has to be auto-create -->
         <address-setting match="activemq.management#">
            <dead-letter-address>DLQ</dead-letter-address>
            <expiry-address>ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <!-- with -1 only the global-max-size is in use for limiting -->
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <auto-create-queues>true</auto-create-queues>
            <auto-create-addresses>true</auto-create-addresses>
            <auto-create-jms-queues>true</auto-create-jms-queues>
            <auto-create-jms-topics>true</auto-create-jms-topics>
         </address-setting>
         <!--default for catch all-->
         <address-setting match="#">
            <dead-letter-address>DLQ</dead-letter-address>
            <expiry-address>ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <!-- with -1 only the global-max-size is in use for limiting -->
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <auto-create-queues>true</auto-create-queues>
            <auto-create-addresses>true</auto-create-addresses>
            <auto-create-jms-queues>true</auto-create-jms-queues>
            <auto-create-jms-topics>true</auto-create-jms-topics>
         </address-setting>
         <address-setting match="exampleQueue">            
            <dead-letter-address>DLQ</dead-letter-address>                      
            <redelivery-delay>1000</redelivery-delay>    
            <max-delivery-attempts>3</max-delivery-attempts>
            <max-size-bytes>-1</max-size-bytes>
            <page-size-bytes>1048576</page-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
        </address-setting>
      </address-settings>
   </core>
</configuration>
1
high-availabilityactivemq-artemis
I have master 192.168.2.111 und two slaves 192.168.2.112 and 192.168.2.113 – LDropl
I would like to use both actually, but need to get work at the beginning at least one of them: between brokers for instance – LDropl
yes self signed certificates – LDropl
Typically when using self-signed certificates the acceptor will have a keystore configured (in order to actually use the self-signed cert) and the connector will have a truststore configured (in order to trust the acceptor's self-signed cert). However, you have both keystore and truststore configured for both the acceptor and connector. Can you clarify why? – Justin Bertram
Does each broker have its own self-signed certificate or are you using the same certificate on all the brokers? – Justin Bertram

1 Answers

0
votes

Connectors have two functions. Defining how to connect to other AMQ brokers and the connector parameters are send to a client when it connects the first time. If a client is going to use these parameters the keystore/truststore path, name and password must exist and be the same on all clients. Best is not to set the key/truststore parameters on a connector then a proper client will use the values from org.apache.activemq.ssl.* or javax.net.ssl.*.

See https://access.redhat.com/documentation/en-us/red_hat_amq/7.4/html/configuring_amq_broker/acceptor_connector_params

I am still struggling to get this to work. AMQ broker HA cluster works without ssl AMQ broker works with ssl AMQ broker HA cluster still fails.

Truststore was a JKS and PKCS12 was expected. Now it works.