Cassandra : data not replicated on new node

Question

I added a new node to my cassandra cluster (the new node is not a seed node). I now have 3 nodes on my cluster :

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               
Rack

UN  XXX.XXX.XXX.XXX  52.25 GB   256          100.0%            XXX  rack1

UN  XXX.XXX.XXX.XXX  63.65 GB   256          100.0%            XXX  rack1

UN  XXX.XXX.XXX.XXX  314.72 MB  256          100.0%            XXX  rack1

I have a replication factor of 3 :

DESCRIBE KEYSPACE mykeyspace

CREATE KEYSPACE mykeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}  AND durable_writes = true;

but the data is not replicated on the new cluster (node with 314 MB of data).

I tried to use nodetool rebuild :

ERROR [STREAM-IN-/XXX.XXX.XXX.XXX] 2016-11-11 08:28:42,765 StreamSession.java:520 - [Stream #0e7a0580-a81b-11e6-9a1c-6d75503d5d02] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-3.1.1.jar:3.1.1] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] ERROR [Thread-16] 2016-11-11 08:28:42,765 CassandraDaemon.java:195 - Exception in thread Thread[Thread-16,5,RMI Runtime] java.lang.RuntimeException: java.lang.InterruptedException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-18.0.jar:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-3.1.1.jar:3.1.1] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74] Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[na:1.8.0_74] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[na:1.8.0_74] at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) ~[na:1.8.0_74] at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:184) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.1.1.jar:3.1.1] ... 1 common frames omitted INFO [STREAM-IN-/XXX.XXX.XXX.XXX] 2016-11-11 08:28:42,805 StreamResultFuture.java:182 - [Stream #0e7a0580-a81b-11e6-9a1c-6d75503d5d02] Session with /XXX.XXX.XXX.XXX is complete WARN [STREAM-IN-/XXX.XXX.XXX.XXX] 2016-11-11 08:28:42,807 StreamResultFuture.java:209 - [Stream #0e7a0580-a81b-11e6-9a1c-6d75503d5d02] Stream failed ERROR [RMI TCP Connection(14)-127.0.0.1] 2016-11-11 08:28:42,808 StorageService.java:1128 - Error while rebuilding node org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.1.1.jar:3.1.1] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:525) ~[apache-cassandra-3.1.1.jar:3.1.1] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279) ~[apache-cassandra-3.1.1.jar:3.1.1] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]

I also tried to change the option but the data is still not copied to the new node : auto_bootstrap: true

Could you please help me understand why the data is not replicated on the new node ?

Please let me know if you need further information from my configuration.

Thank you for your help

Check out stackoverflow.com/questions/16343142/… first- if the answers there don't apply we can dig deeper. — Chris Shain
Thanks Chris for your help. I actually read this question first. I have the same num_tokens on the 3 nodes (same hardware config). I added auto_bootstrap=true in the cassandra config. I ran nodetool repair on the new node , nodetool cleanup on the other nodes but the data would not be replicated on the new node. — Bertrand

Chris Shain Chris Shain · Accepted Answer · 2016-11-11T16:23:04

It appears (from https://issues.apache.org/jira/browse/CASSANDRA-10448) that this is due to CASSANDRA-10961. Applying that fix should address it.

Cassandra : data not replicated on new node

1 Answers