How to resolve the conflict between 11.0.2 and 16.0 of guava when using yarn, spark and spark-cassandra-connector?

Question

my yarn's version is hadop-2.4.0.x, spark is spark-1.5.1-bin-hadoop2.4 and spark-cassandra-connector is spark-cassandra-connector_2.10-1.5.0-M2, when I executed the following command:

bin/spark-shell --driver-class-path $(echo lib/*.jar | sed 's/ /:/g')  --master yarn-client 
--deploy-mode client --conf spark.cassandra.connection.host=192.21.0.209 
--conf spark.cassandra.auth.username=username --conf spark.cassandra.auth.password=password --conf spark.sql.dialect=sql 
--jars lib/guava-16.0.jar,spark-cassandra-connector_2.10-1.5.0-M2.jar,lib/cassandra-driver-core-2.2.0-rc3.jar

After starting, I input the following scala under the prompt:

import org.apache.spark.sql.cassandra.CassandraSQLContext
import org.apache.spark.sql.{DataFrame, SaveMode}
import org.apache.spark.{Logging, SparkConf, SparkContext}
import org.joda.time.{DateTime, Days, LocalDate}
val cc = new CassandraSQLContext(sc)

val rdd: DataFrame = cc.sql("select user_id,tag_models,dmp_province," +
"zp_gender,zp_age,zp_edu,stg_stage,zp_income,type " +
"from user_center.users_test")

I got the classic error:

Caused by: java.lang.NoSuchMethodError:  
com.google.common.util.concurrent.Futures.withFallback
(Lcom/google/common/util/concurrent/ListenableFuture;
Lcom/google/common/util/concurrent/FutureFallback;
Ljava/util/concurrent/Executor;)
Lcom/google/common/util/concurrent/ListenableFuture;

After search this error in google and stackoverflower, I know that the conflict between the different versions of guava caused this error, and found hadoop 2.4 use guava-11.0.2 but spark-cassandra-connector_2.10-1.5.0-M2 use guava-16.0.1.

How to resolve this kind of error, any advice will be appreciated!

UPDATE

I am so sorry for testing long time!

Now, for spark-submit, I tested this resolution Making Hadoop 2.6 + Spark-Cassandra Driver Play Nice Together successfully under my test yarn cluster

My man, you are on the track. If you happen to found the answer please share it with us. I think the road is to shade at sbt assembly stage. — msemelman
@msemelman I am sorry, I did not come here yesterday, Do you have the same issue? — abelard2008
@msemelman, I tested the approach as #user1314742 said, nothing changed! — abelard2008
@abelard2008 I added new comment to my answer, please try to the export mentioned in the comment. — user1314742
@abelard2008 one more thing, in your answer is it $(echo lib/*.jar | sed 's/ /:/g') or you meant $(ls lib/*.jar | sed 's/ /:/g') ? — user1314742

user1314742 user1314742 · Accepted Answer · 2016-03-15T21:04:25

In Hadoop configuration, add the following property to your hadoop-env.sh

HADOOP_USER_CLASSPATH_FIRST=true

In Spark configurations there is also a property you could set it to true spark.driver.userClassPathFirst , but it still an experimental and used only in cluster mode (have a look at the spark documentation). Personally I have not tried this property but since it is presented in the documentation I thought it is worth to mention

How to resolve the conflict between 11.0.2 and 16.0 of guava when using yarn, spark and spark-cassandra-connector?

1 Answers