1
votes

I have a Spark Ignite program. When I use sbt to build the program, it reports error.

This is a very simple program, which is an empty project with just a sbt file.

build.sbt:

name := "testsss"

version := "1.0"

scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
libraryDependencies += "org.apache.ignite" % "ignite-core" % "1.9.0"
libraryDependencies += "org.apache.ignite" % "ignite-indexing" % "1.9.0"
libraryDependencies += "org.apache.ignite" % "ignite-spark" % "1.9.0"

The error is

Error:Error while importing SBT project:
...

[info]
Resolving org.scala-sbt#testing;0.13.8 ... [info] Resolving
org.scala-sbt#test-agent;0.13.8 ... [info] Resolving
org.scala-sbt#test-interface;1.0 ... [info] Resolving
org.scala-sbt#main-settings;0.13.8 ... [info] Resolving
org.scala-sbt#apply-macro;0.13.8 ... [info] Resolving
org.scala-sbt#command;0.13.8 ... [info] Resolving
org.scala-sbt#logic;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_8_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_3;0.13.8 ... [error] Modules were
resolved with conflicting cross-version suffixes in
{file:/Users/lzhan71/Documents/IgniteMaster/}ignitemaster: [error]
org.scalatest:scalatest _2.11, _2.10 [error] com.twitter:chill _2.11, _2.10 [error] org.apache.spark:spark-unsafe _2.11, _2.10 [error] org.apache.spark:spark-tags _2.11, _2.10 [trace] Stack trace suppressed: run 'last *:update' for the full output. [trace] Stack trace suppressed: run 'last :ssExtractDependencies' for the full output. [error] (:update) Conflicting cross-version suffixes in: org.scalatest:scalatest, com.twitter:chill, org.apache.spark:spark-unsafe, org.apache.spark:spark-tags [error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.scalatest:scalatest, com.twitter:chill, org.apache.spark:spark-unsafe, org.apache.spark:spark-tags [error] Total time: 17 s, completed Apr 28, 2017 4:41:00 PM

See complete log in /Users/lzhan71/Library/Logs/IdeaIC2016.2/sbt.last.log

I checked. It seems that `"ignite-spark" % "1.9.0" used a different version of those conflict dependence with spark-core and spark-sql. How could I solve this?

2

2 Answers

1
votes

This seems to be a bug in ignite-spark. It depends simultaneously on spark-unsafe_2.10 and scala-library 2.11.8, which should never happen. Report it and wait until a new version is released or workaround by getting its source code, fixing the spark-unsafe_2.10 dependency and recompiling (quite likely no source code fixes are needed).

1
votes

You can also try fixing it like this: libraryDependencies += "org.apache.ignite" % "ignite-spark" % "2.2.0" excludeAll(ExclusionRule("org.apache.spark"))