Multi-Region Cassandra in AWS only local shows on “nodetool status”

1
votes

Hope someone had the same problem and is able to give me help. I am trying to create a "3 nodes (AWS DC1) <-> 3 nodes(AWS DC2)" Cassandra cluster. My first step is create a 3 nodes cluster on AWS DC1. "nodetool status" only shows local is up. Settings --- All nodes in a same private subnet. Each node has a public and a private ip. all inbound open to All traffic, all ports, everywhere for testing. cassandra.yaml : listen_address private IP address broadcast_address public IP address endpoint_snitch Ec2MultiRegionSnitch seeds: public IP address

Thank you.

1
amazon-web-servicescassandra
DEBUG [MessagingService-Outgoing-/（public_ip of node1)-Gossip] OutboundTcpConnection.java:546 - Unable to connect to /（public_ip of node1) java.net.ConnectException: Connection timed out ... at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:645) ~[na:1.8.0_242] ..at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:434) [apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-cassandra-3.11.3.jar:3.11.3] DEBUG [MessagingService-Outgoing-/（public_ip of node2)-Gossip] . - dixon Sun

1 Answers

1
votes

Having the public IP address is not needed, neither accept income communication in all the ports, the default ports used by Cassandra are:

  • TCP 7000 - Used for intra-node communication. In cassandra.yaml this is configured in storage_port
  • TCP 7001 - similar to 7000, used when the intra-node SSL encryption is enabled; In cassandra.yaml this is configured in ssl_storage_port
  • TCP 7199 - Used for JMX communication, this is needed by nodetool and could also be used for monitoring
  • TCP 9042 - Used by native clients when encryption is disabled, In cassandra.yaml this is configured in native_transport_port
  • TCP 9142 - Used by native clients when encryption is enabled, In cassandra.yaml this is configured in native_transport_port_ssl

Note that if you have additional tools, like opscenter, more tools will be needed to be opened.

In the cassandra.yaml, the listen_address, broadcast_address, and rpc_address would be better to set it with the private IP address.

For the seed_provider configuration, ensure that the class_name used is org.apache.cassandra.locator.SimpleSeedProvider. When you are starting the cluster from scratch, the first node should have its own private address in the seeds list. Once that the node is UN (Up and Normal), the other nodes should include the list of the private addresses of the other nodes, but not their own IP address. Once that the cluster is up with the 3 nodes you should update the cassandra.yaml of the first node to replace its own address with the list of addresses of the other nodes.

Once that you have the cluster on DC1, the first node of DC2 should include at least an IP address of DC1, that node should be running.

Final note: remember to do all this process one node at a time.