Cassandra in AWS Multi Region VPC (VPC peering)

Question

We are trying to setup following cluster geography for Cassandra:

2 VPCs in 2 different AWS regions (US-West, APAC-Singapoore)
VPCs connected via VPC peering (common IP space)
2 Cassandra nodes in each DC, 1 DC in each region (4 nodes total)
Nodes communicating together with private IP in VPC
Connection to nodes should be available only inside VPC, no Public IPs!

Using Gossiping Snitch
Looks like nodes in one region are able to see second region DC, but not the other way around. Spend some time and cannot find the issue.

What configuration setup would you recommend? Maybe good advice how to troubleshoot possible network misconfigurations
Node 1 (us-west-2a - 10.200.0.100)

cassandra.yaml
broadcast_address: 10.200.0.100
listen_address: 10.200.0.100
rpc_address: 0.0.0.0
broadcast_rpc_address: 10.200.0.100

Nodetool output on DC1 node 1 (us-west-2a)

Datacenter: ap-southeast
    Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID       Rack 
DN  10.201.1.77   97.02 KB   256          ?       ac72b936-8c5c-41db-ad06-fa056759e704  1a
DN  10.201.1.93   141.97 KB  256          ?       78c4b0c5-adff-40b8-913d-8be063454643  1a

Datacenter: us-west-2a

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID    Rack 
UN  10.200.0.52   671.59 KB  256          ?       b84f2113-90ef-43d4-91df-eed3fbb6ff79  2a
UN  10.200.0.100  516.96 KB  256          ?       3ce8d7ea-d586-48f8-9f8a-b9a5c1fb8fa0  2a

Node 3 (ap-southeast - 10.201.1.93)

cassandra.yaml
broadcast_address: 10.201.1.93
listen_address: 10.201.1.93
rpc_address: 0.0.0.0
broadcast_rpc_address: 10.201.1.93

Nodetool output for DC2 node 3 (ap-southeast)

Datacenter: ap-southeast

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns    Host ID   Rack 
UN  10.201.1.93  136.2 KB   256          ?       78c4b0c5-adff-40b8-913d-8be063454643  1a

Can you compare the route table entries for the subnet in both regions? Do you see they are setup identically? — helloV
Did you use the external IP's for all the nodes in both DC's? Can you post the output of a nodetool status command. — bechbd
@bechbd I've added nodetool output. There no external IPs as we are in VPC. I've used privite IPs for all machines. By IPs nodes can resolve each other — Anton G
@TonyG; Can you post the broadcast_address, listen_address, and rpc_address from a cassandra.yaml file in each DC? Also which DC is working correctly and which one isn't? — bechbd
@bechbd I've added cassandra.yaml listings from nodes 1, 3 in different DC. Can add more info if this helps — Anton G

Ayush Sharma Ayush Sharma · Accepted Answer · 2017-03-24T00:14:30

Did you try using a VPN to connect the two? I don't think VPC Peering works across multiple regions. It's only for connecting VPCs within the same region.

https://notes.ayushsharma.in/2017/02/connecting-vpcs-in-2-aws-regions-site-to-site-vpn

Cassandra in AWS Multi Region VPC (VPC peering)

1 Answers