Conflicting cross version suffixes in: Spark job

Question

Below is my build.sbt file. I have searched other questions like this but none have helped me find an answer to this. I have tried multiple ways to explicitly use 2.11 scala but for some reason I keep getting this error.

[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in:

org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-core, org.apache.spark:spark-network-common

[error] (*:update) Conflicting cross-version suffixes in: org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-core, org.apache.spark:spark-network-common

name := "ubi-journeyData-validation"

version := "2.0"

scalaVersion := "2.11.11"
dependencyOverrides += "org.scala-lang" % "scala-compiler" % scalaVersion.value

//updateOptions := updateOptions.value.withCachedResolution(false)

libraryDependencies ++= {
  val sparkVersion = "2.3.0"
  //val sparkVersion = "1.6.3"
     Seq("org.apache.spark" %% "spark-core" % sparkVersion,
    "org.apache.spark" %% "spark-sql" % sparkVersion,
    "org.apache.spark" %% "spark-hive" % sparkVersion,
    "org.elasticsearch" %% "elasticsearch-spark-20" % "5.6.9",
    //"org.elasticsearch" %% "elasticsearch-spark-13" % "5.6.9",
    //"org.elasticsearch" %% "elasticsearch-spark-13" % "5.2.0",
    "org.spark-project.hive" % "hive-cli" % "1.2.1.spark2",
    "org.spark-project.hive" % "hive-metastore" % "1.2.1.spark2",
    "org.spark-project.hive" % "hive-exec" % "1.2.1.spark2"
    //"org.json4s" %% "json4s-jackson" % "3.2.11",
    //"org.apache.calcite" % "calcite-core" % "1.2.0-incubating",
    //"org.pentaho" % "pentaho-aggdesigner" % "5.1.5-jhyde" pomOnly(),
    //"org.pentaho" % "pentaho-aggdesigner-algorithm" % "5.1.5-jhyde" % Test
   )
}

resolvers += Resolver.mavenLocal
resolvers += "Cascading repo" at "http://conjars.org/repo"


assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

Atais Atais · Accepted Answer · 2018-08-30T18:33:22

It is because of dependencies of one of your dependency, fe:

"org.spark-project.hive" % "hive-exec" % "1.2.1.spark2"

which if you check, for example here, it depends on:

org.apache.spark » spark-core_2.10 » 1.3.1 » (optional)

so I am afraid you can't update your spark to 2.3.0 and Scala to 2.11 unless you dump those org.spark-project.hive modules.

Conflicting cross version suffixes in: Spark job

1 Answers