How to make zone/dc awareness connection to Cassandra from spark?

1
votes

I am using spark-sql 2.4.1 , spark-cassandra-connector_2.11-2.4.1.jar and java8. While I am trying to fetch data from table I am encountering 

java.io.IOException: Failed to write statements to keyspace1.model_vals. The
latest exception was
  An unexpected error occurred server side on cassandra-node1: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.

So how to make zone/dc awareness connection to Cassandra db from spark code ?

YML

existing one

spring:
  data:
      cassandra:
        keyspace-name: raproduct
        contact-points:
                    - cassandra-node1
                    - cassandra-node2
        port: 9042

Changed to

spring:
  data:
      cassandra:
        connection:
          local_dc: southeast-1
        keyspace-name: raproduct
        contact-points:
                    - cassandra-node1
                    - cassandra-node2
        port: 9042

Question

But it is not reflected/applied the changed "local_dc". How to do it in spring-data ?

1
apache-sparkcassandraspring-datadatastax-java-driverspring-data-cassandra

1 Answers

5
votes

Check through the Spark Connector documentation and in the Configuration Reference - Cassandra Connection Parameters. It would seem that this can be done by setting the spark.cassandra.connection.local_dc property in the connection configuration:

val conf = new SparkConf(true)
        .set("spark.cassandra.connection.host", "192.168.1.10")
        .set("spark.cassandra.auth.username", "flynn")            
        .set("spark.cassandra.auth.password", "reindeerFlotilla82")
        .set("spark.cassandra.connection.local_dc", "encom_west1_dc")

val sc = new SparkContext("spark://192.168.1.133:7077", "test", conf)

Not sure what your connection config code looks like, but try setting that spark.cassandra.connection.local_dc property and see where that gets you.