an error in spark sbt assembly

1
votes

In spark installing README,I write command './bin/spark-shell', I got some hint follwing:

Found multiple Spark assembly jars in /opt/spark-0.9.0-incubating-bin-hadoop1/assembly/target/scala-2.10: spark-assembly-0.9.0-incubating-hadoop1.0.4.jar spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar Please remove all but one jar.

I ever try to remove one of them ,but I fail to success. If somebody can deal with it ,please tell me. thanks.

2
scalaapache-spark

2 Answers

0
votes

all you need to do is just cd to the assembly/target/scala-2.10 dir, and use the sbt to compile again.

this is caused by the incompatible between different compile method, you may used the maven and sbt both?

0
votes

The problem is that the JARs are built into the assembly JARs when you run

sbt/sbt clean assembly

What I did is: sbt/sbt clean package

This will only give you the small JARs. The next steps is to update the CLASSPATH in the bin/compute-classpath.sh script manually, appending all the JARs.

With :

sbt/sbt assembly

We can't introduce our own Hadoop patch since it will always pull from Maven repo, unless we hijack the repository path, or do a 'mvn install' locally. This is more of a hack I think.