We are facing several problems during commissioning of new nodes to the existing cluster. The existing cluster(5 nodes) is holding data of 13 TB and daily 0.1 TB of data will be loaded.Ten days back,we started adding 5 nodes. In the middle of the commissioning process, the bootstrap process is getting failed many times, displaying an error STREAM FAILED in the new node and BROKEN PIPE in the old node. Whenever we face this problem, we are restarting the new node. Heare are the few questions about the node joining process.
1) Whenever we are restarting the node to join, will the bootstrap process resume or restart from the begining? If so, should we wipe off the data directories and do a fresh restart.
2) How much time it might take to complete the node joining process.( Network bandwidth :1Gbps) ?
3) Can we add a node directly by setting auto_bootstrap to false and run a nodetool repair in that node?
4) How to monitor the percentage of load balancing in each node?
5) Can we increase the streaming speed by using nodetool setstreamthroughput property? What is the difference between stream_throughput_outbound_megabits_per_sec property in cassandra.yaml file and nodetool setsreamthroughput property?
6) Can we scp some data from old nodes to new nodes and do a restart?
