1
votes

I am adding nodes to my existing cassandra cluster. Bootstrapping got stuck after Node streams all the data. netstats shows no data in stream to new node. When I check the nodetool - info for new node it shows an exception. This exception could be the reason bootstrapping gets stuck.

Log file on joing node shows the following exception. But following exception did not stop the streaming. This node streams the data after exception too.

java.io.EOFException: EOF after 129546466 bytes out of 825307185 at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:229) at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:50) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:451) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1101) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1092) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

2

2 Answers

1
votes

I think you are right that this is why bootstrap got stuck.

I'd run "nodetool scrub" against the existing nodes, then retry the bootstrap on the new one (after clearing out the data from the first attempt).

0
votes

Just adding above answer.Even the nodetool scrub didn't worked and some of the nodes were still giving problems in our case.

Upgrading to 1.1 and loading data with sstable loader solved the issue. I can't comment whether this is optimum solution but it worked in our case.