Play 2.x with Cassandra Java Driver - ERROR com.datastax.driver.core.Session - Error creating pool to xxx.xx, but return responses

Question

Set up

Cassandra Java Driver : 2.1.3, 2.1.4, 2.0.0 (tried with all)
Cassandra Cluster : [cqlsh 5.0.1 | Cassandra 2.1.0 | CQL spec 3.2.0 | Native protocol v3]
Play 2.2.3

I get below issue with above set up while accessing a simple table. But I also get responses as in the log. I am running servers on EC2, 2 node single datacenter. Also tried with EC2 AMI but same results. Further this is a very fresh set up and no load on the servers yet.

Cassandra Configs

rpc_address : 0.0.0.0
listen_address : internal_ip
broadcast_address: public_ip
start_native_transport: true
native_transport_port: 9042

Code snippet

        SocketOptions options = new SocketOptions();
        options.setConnectTimeoutMillis(10000);
        options.setReadTimeoutMillis(10000);

        cluster = Cluster.builder().addContactPoints("public_ip1","public_ip2").withPort(9042).withSocketOptions(options).withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)
                .withReconnectionPolicy(new ConstantReconnectionPolicy(1000L)).build();

         Metadata metadata = cluster.getMetadata();
         System.out.printf("Connected to cluster: %s\n", 
                metadata.getClusterName());

        session = cluster.connect();

        // Use select to get the user we just entered
        ResultSet results = session.execute("SELECT * FROM redmart.user_pick_rate");
        for (Row row : results) {
            System.out.println( "user_id : " + row.getInt("user_id") + " pick_rate : " + row.getDouble("pick_rate"));
        }

According to the below exception, it always complains "Error creating pool to /internal_ip:9042" to one of the nodes in Java client. example if I have node1, node2. In the javaclient if I create a cluster only with node1, it complains cannot create the pool to (internal ip of node2), if I use node 2 to create cluster, it it complains cannot create the pool to (internal ip of node1). If I specify both it always says Error creating pool to one of the nodes. I tried with different netty versions too.

I saw similar unresolved issue in here, but in my case I am getting responses as well.

Any insights are appreciated.

11:59:31.791 [Cassandra Java Driver worker-0] ERROR com.datastax.driver.core.Session -     Error creating pool to /10.167.135.180:9042
com.datastax.driver.core.TransportException: [/10.167.135.180:9042] Cannot connect

at com.datastax.driver.core.Connection.<init>(Connection.java:104) ~     
at com.datastax.driver.core.SingleConnectionPool.<init>(SingleConnectionPool.java:76) ~[cassandra-driver-core-2.1.4.jar:na]
at   com.datastax.driver.core.HostConnectionPool.newInstance(HostConnectionPool.java:35) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:271) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:40) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:308) [cassandra-driver-core-2.1.4.jar:na]
at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:300) [cassandra-driver-core-2.1.4.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /10.167.135.180:9042
at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[cassandra-driver-core-2.1.4.jar:na]
at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[cassandra-driver-core-2.1.4.jar:na]
... 3 common frames omitted
user_id : 6555 pick_rate : 0.4494318731691269
user_id : 1000 pick_rate : 7.132659816872288E-4

I am having this same problem. Play connects, I can insert into the DB etc, but this exception is still thrown. Did you ever find out the answer? — plamb
Actually not, I have posted the same question to datastax-Java community. datastax-oss.atlassian.net/browse/JAVA-535 — Harsha
I'm having the same issue right now. It seems that the Client is trying to connect to the nodes through the EC2 internals IPs :( — Eduardo Pascual Aseff

Arodrime Arodrime · Accepted Answer · 2015-02-25T15:49:29

Try with something else than rpc_address : 0.0.0.0.

Off the top of my head this is blocking for native clients. Try instead with the private IP (not sure it was private IP and not public, you can test both though).

Play 2.x with Cassandra Java Driver - ERROR com.datastax.driver.core.Session - Error creating pool to xxx.xx, but return responses

1 Answers