setting up cassandra multi node cluster on a single ubuntu server

6
votes

I have a Cassandra Service running on my Ubuntu Server with a single node now. I want to make it into a ring cluster with 3 nodes to get a feel of multinode cluster all being on the same server. By following the steps in this link https://www.youtube.com/watch?v=oHMJrhMtv3c, I tried to create a fresh cluster without stopping the already running cassandra service. But it has thrown address Caused by: java.net.BindException: Address already in use. So i tried changing the seeds ip to already running cassandra ip address and tried to run a second cassandra service in the foreground. This time it has thrown java.lang.RuntimeException: Unable to create thrift socket to ip port. Please let me know how to add nodes to a already running single node cluster on the same server.

5
ubuntucassandracluster-computing

5 Answers

7
votes

It is very easy to run multiple Cassandra instances on the same Ubuntu machine as long as they have different IP addresses that all resolve to the local host. The entire 127.0.0.0/8 address block is reserved for loopback purposes so any packet sent to addresses (127.0.0.1 through 127.255.255.254) will be looped back.

  • Use ping to check if the addresses resolve properly.
  • Place two (or more) Cassandra instances into different folders.

Edit cassandra.yaml and replace

  • Various file locations to the locations unique to the given instance of Cassandra.
  • localhost to the IP address we give to that instance (like 127.0.0.2).
  • Use SimpleSeedProvider and put addresses of all other Cassandra instances to the seed list to make a cluster (like - seeds: "127.0.0.2","127.0.0.3"
  • Do not alter any port numbers, not helpful and not required.

Edit cassandra-env.sh, find where the JMX_PORT property is set and give it a different value (different port) for every instance of Cassandra. Otherwise instances cannot run together because of the conflicts on this port.

  • start the instances one by one using ./cassandra startup script (you can write simple bash script for this later).

  • Verify your topology with ./nodetool status . For the two nodes, for instance, the output must look like

    Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
UN  127.0.0.1  61.97 KB   256     100.0%            6c04e202-8f24-4f17-b430-0154c1512316  rack1
UN  127.0.0.2  105.68 KB  256     100.0%            ca3073ee-451c-4cef-97ee-d312784648bb  rack1
4
votes

The easiest way to set up a multinode cluster on a single machine is using CCM. Currently you are running into issues with your nodes attempting all bind the same set of ports. CCM will work around this for you and auto increment ports ect...

https://github.com/pcmanus/ccm

0
votes

Tried this on,

Cassandra: 3.11.2
RHEL: 6.5

In cassandra.yaml, keep the following values different for different instances:

  1. native_transport_port
  2. Data Directory
  3. Commitlog Directory
  4. Saved_Cache Directory
  5. listen_address
  6. broadcast_rpc_address(rpc_address set as 0.0.0.0)

Use loopback address in both listen_address and broadcast_rpc_address. Keep the storage port same in all the instances(i.e. 7000).

In cassandra-env.sh, keep the JMX_PORT different for different instances.

0
votes

Elaborating on the answer provided by h22 (I cannot comment on it yet) You will need node0 to be running on localhost with the default jmx port. It's only the additional nodes that need to change the cassandra.yaml Further if you are using cassandra 3 or beyond (on windows?), one needs to start the second cassandra node with -a parameter to skip the port checks.

0
votes

ccm does the whole thing automatically within seconds...

$ccm create -v 3.11.11 -n 3 my_cluster --vnodes

This creates 3 different folders for the 3 instances + 1 folder to the common binaries.

$ccm status

Cluster: 'my_cluster'
---------------------
node1: DOWN (Not initialized)
node3: DOWN (Not initialized)
node2: DOWN (Not initialized)


$ccm start
$ccm status
Cluster: 'my_cluster'
---------------------
node1: UP
node3: UP
node2: UP


$ccm node1 status
Datacenter: datacenter1
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  127.0.0.1  142.01 KiB  256          63.6%             8e48c1fd-656d-4e9c-b3b0-ae92f0d639b3  rack1
UN  127.0.0.2  172.07 KiB  256          70.9%             fcd79431-1f64-4021-9fe4-0ec6630b234b  rack1
UN  127.0.0.3  172.3 KiB  256          65.5%             3938cef1-9f62-4143-b15d-c7f32c83d76b  rack1