Error: Must specify a primary resource (JAR or Python file) - Spark scala

4
votes

I am trying to run a simple twitter sentiment analysis code that was running fine upto now but I don't know what changed which is giving me this error. My command line has all the required parameters in place including --class --master --jars etc. The only thing I did different was run the sudo apt-get install 7-jdk command and that updated the java version. I am running spark 1.3.1 so this java updation should not be a problem...I think. Now even when I run commands like sbt assembly or sbt run I get an error saying assembly is not a know command.

Here's my command line:

./bin/spark-submit --class Sentimenter --master local[4] --jars /home/ubuntu/spark/spark-example-master/target/scala-2.10/Sentiment_Analysis-assembly-1.0.jar

And here's the output I get:

Error: Must specify a primary resource (JAR or Python file) Run with --help for usage help or --verbose for debug output Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Any suggestions would be great!

2
scalaapache-spark
Here's a suggestion: Add the exact command you typed and the exact output you get. Then people might be able to help you. - Iulian Dragos
So I was able to fix this problem to some extent by using SPARK_JAVA_OPTS=-Dlog4j.configuration=log4j.properties but now I get a different error saying: ERROR SparkContext: Jar not found at file:/home/ubuntu/spark/SPARK_JAVA_OPTS=-Dlog4j.configuration=log4j.properties - serendipity

2 Answers

9
votes

From spark-submit --help:

Usage: spark-submit [options] <app jar | python file> [app arguments]

You need to add a jar or python file on the command line, after options. So, in your example, you'd need to do

./bin/spark-submit --class Sentimenter --master local[4] /home/ubuntu/spark/spark-example-master/target/scala-2.10/Sentiment_Analysis-assembly-1.0.jar

Note that I removed --jars. You can do that to add additional jars (dependencies). If all you have is one jar, that's the primary resource and goes on the command line without any --jars option.

0
votes

this might sounds ridiculous. but this is my situation.

I had extra long command line, so each parameter is broken into different lines in order to be readable.

when there are too many empty spaces in the argument, I had exactly this error reported. so I removed unnecessary spaces and the error go away.

if you have followed above suggestion and there are still weird errors, take a look at your extra spaces.