sstableloader fails to upload data if node is down

Question

I have a test setup in which sstableloader fails to upload data if one of the Cassandra node is down. I can see that

Is there a way to instruct sstableloader not to connect (or open a stream) to the dead node (I don't want to decommission/remove the node from cluster)?

Cassandra cluster info: Datastax community version 2.1.2, 3 node cluster out of which 2 are seed nodes.

During testing bulk upload, one of the seed node was down. The keyspace has replication factor = 2.

Exception encountered:

progress: total: 100% 0  MB/s(avg: 0 MB/s)ERROR 09:07:48 [Stream #8972f510-efe1-11e4-abad-9d409520f182] Streaming error occurred
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_65]
        at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_65]
        at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_65]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_65]
        at java.nio.channels.SocketChannel.open(SocketChannel.java:184) ~[na:1.7.0_65]
        at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:229) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:79) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:216) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:208) [apache-cassandra-2.1.2.jar:2.1.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
progress: [/192.168.1.17]0:1/1 100% total: 100% 0  MB/s(avg: 1 MB/s)WARN  09:07:48 [Stream #8972f510-efe1-11e4-abad-9d409520f182] Stream failed
Streaming to the following hosts failed:
[/192.168.1.15]
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:121)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:382)
        at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:574)
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:438)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:251)
        at java.lang.Thread.run(Thread.java:745)

Thanks in advance, Anirban.

Anirban Anirban · Accepted Answer · 2015-05-01T12:39:23

I just figured that I can pass an ignore list to sstableloader. By passing dead nodes in the ignore list sstableloader ran successfully in my test setup.

sstableloader fails to upload data if node is down

1 Answers