DSE Cassandra has guava-16.0.1.jar conflict issue with CDH spark

1
votes

We used DSE4.8.3 Cassandra to run CDH5.5.0 Spark in oozie, just found that DSE Cassandra has guava-16.0.1.jar conflict issue as following.

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, com.google.common.reflect.TypeToken.isPrimitive()Z java.lang.NoSuchMethodError: com.google.common.reflect.TypeToken.isPrimitive()Z

Cassandra version in DSE 4.8.3 was 2.1.11.969. Spark version in CDH 5.5.0 was 1.5.0. For cassandra driver and connector.

1.If we used cassandra-driver-core-2.2.0-rc3.jar and spark-cassandra-connector_2.10-1.5.0-M2.jar, which both used guava-16.0.1.jar as their dependencies, it threw above exception "Method not found: com.google.common.reflect.TypeToken.isPrimitive()Z" in CDH (CDH5.5.0 spark used guava-14.0.1.jar, not guava-16.0.1.jar ).

2.If we used lower version cassandra-driver-core-2.2.0-rc1.jar and spark-cassandra-connector_2.10-1.5.0-M1.jar, which both used guava-14.0.1.jar as their dependencies, it threw following exception: Exception in thread "main" java.lang.AbstractMethodError: com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.close()V

at com.datastax.driver.core.Cluster$Manager.close(Cluster.java:1417)

at com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:1167)

at com.datastax.driver.core.Cluster.closeAsync(Cluster.java:461)

at com.datastax.driver.core.Cluster.close(Cluster.java:472)

at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:163)

I found an answer for this exception:(saying that using upper version spark-cassandra-connector_2.10-1.5.0-M2.jar will resolve the issue) Spark + Cassandra connector fails with LocalNodeFirstLoadBalancingPolicy.close()

So now, we are mystified with the Cassandra dependencies issue. How to fix this cassandra guava-16.0.1 dependency issue? Is it possible to build a new spark-cassandra-connector.jar fixing with both issues? Can you help to resolve this issue? Thanks!

1
apache-sparkcassandraooziedatastax-enterprise

1 Answers

2
votes

There should be no C* Driver dependency as that should be brought in automatically with the Spark Cassandra Connector Dependency as a transitive dependency. I would use the 1.5.0 release. Then you need to make sure when building that you exclude all other guava versions.

This means make sure if you are making a fat jar you aren't including any Spark distributions in your code and any Hadoop libs have Guava excluded.

There are a few other mail threads on this for more details

Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use. https://groups.google.com/a/lists.datastax.com/forum/#!topic/spark-connector-user/HnTsWJkI5jo

Issue with guava https://groups.google.com/a/lists.datastax.com/forum/#!topic/spark-connector-user/uB_DN_CcK2k