I have a 3 cassandra nodes (db-stats-01, db-stats-02, db-stats-03), which are all Up and Normal according to nodetool status.
The client is essentially doing this:
Builder builder = Cluster.builder();
builder.addContactPoint(node);
context = getSSLContext();
String[] cipherSuites = SSLOptions.DEFAULT_SSL_CIPHER_SUITES;
builder.withSSL(new SSLOptions(context, cipherSuites)).build();
and failing on the last line with the following exception:
Sep 12 14:31:26 localhost daemon: Defuncting connection to db-stats-01/192.168.105.1
Sep 12 14:31:26 localhost com.datastax.driver.core.ConnectionException: [db-stats-01/192.168.105.1] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [db-stats-01/192.168.105.1] Operation Timeouted)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection.initializeTransport(Connection.java:182)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection.<init>(Connection.java:132)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection.<init>(Connection.java:60)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection$Factory.open(Connection.java:419)
Sep 12 14:31:26 localhost at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:205)
Sep 12 14:31:26 localhost at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:168)
Sep 12 14:31:26 localhost at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:81)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:794)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:721)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster.<init>(Cluster.java:82)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster.<init>(Cluster.java:67)
Sometimes, but not everytime, I get the following error on the server side:
DEBUG [New I/O worker #1] 2014-09-12 14:36:14,337 Slf4JLogger.java (line 36) SSLEngine.closeInbound() raised an exception after a handshake failure.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1587)
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1517)
at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1407)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1293)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:913)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DEBUG [New I/O worker #1] 2014-09-12 14:36:14,342 Slf4JLogger.java (line 36) Swallowing an exception raised while writing non-app data
java.nio.channels.ClosedChannelException
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at org.jboss.netty.channel.Channels.write(Channels.java:686)
at org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:1153)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1246)
at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:656)
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:361)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
However, most of the time I don't even see this error on the server, dispite the server being set to DEBUG.
I don't suspect a connectivity issue, because I have watched tcpdump and I see packets going to and from the client and server. Also, there is briefly an ESTABLISHED connection up on port 9042 when I look at netstat -anp.
Using telnet, I can verify that the server is listening on 9042, and when I send junk through telnet I get a nice error on the server side complaining about a non-SSL record:
ERROR [Native-Transport-Requests:85] 2014-09-12 14:40:23,747 ErrorMessage.java (line 222) Unexpected exception during request
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 640d0a64660d0a
Not sure what else to check or where I may have gone wrong. Before implementing SSL in my client, I didn't see this issue at all, so my thought is it is probably related. Other misc info:
[root@db-stats-01 ~]# rpm -qa | grep cassandra
cassandra20-2.0.9-1.noarch
[root@db-stats-01 ~]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
