Does spark-cassandra-connector support built-in load balanceing?

Question

I have Scala-based application and I need to connect it to Cassandra. I found DataStax Enterprise drivers very useful in this regard, and those have a lot of cool features like in-built load balancing for Cassandra and that is really import for me. Unfortunately there isn't any native DSE drivers for Scala. I know we can use DSE Java drivers, but in that case, we loose a lot of Scala cool features. I also found spark-cassandra-connector that's built by Datastax as well, but this built-in load balancing thing is really important to me and I don't know if spark-cassandra-connector support it or not.

In the Java-based applications using DSE Java driver, I need to config the built-in load balancer in a configuration file as below:

datastax-java-driver.basic.load-balancing-policy {
  class = DefaultLoadBalancingPolicy
}

I don't know the equivalent way in Scala using spark-cassandra-connector and I'm not even sure if it is possible or not. Any help would be appreciated. Thanks.

Answer 1

In the Scala you can just use the Java driver - out of the box you don't have only support for base Scala types, but you can solve this problem by importing the java-driver-scala-extras into your project (as source code) - it works for at least for driver 3.x. Another issue is the support for Option, but this could done via Java's optional that has an extra codec in Java driver.

Regarding the customization of the driver - that part should work with Scala without change. Regarding the support of default policy in Spark - Spark Cassandra connector has a separate policy for a special reason - it's close to the Java's default policy, but with specifics for Spark.