spark-submit on ibm bluemix

Question

i've just registrated a free instance of "Analitycs for Apache Spark" and followed this tutorial to use spark submit ibm ad hoc designed script to run an app from my local machine on bluemix cloud cluster. The issue is the following: i've made everithing that was described in the tutorial and lunched this script

./spark-submit.sh --vcap ./vcap.json --deploy-mode cluster --master
 https://spark.eu-gb.bluemix.net --files /home/vito/vinorosso2.csv
 --conf spark.service.spark_version=2.2.0 
/home/vito/workspace_2/sbt-esempi/target/scala-2.11/isolationF3.jar 
--class  progettoSisDis.MasterNode

everything proceed fine (dataset vinorosso2.csv and my fatJar are correctly uploaded) until the terminal sais :" submission complete" at this point when i go to the log file created by the script there was this error message :

Submit job result: Invalid plan and spark version combination in HTTP request (ibm.SparkService.PayGoPersonal, 2.0.0)

Submission ID: ERROR: Problem submitting job. Exit

So, it wasn't enough to register a free instance of Analitycs for apache spark to submit a spark job? Hope someone can help. By the way, if it helps, on my local machine i'm using spark with intellij idea (scala). Byyye

shawdm shawdm · Accepted Answer · 2017-11-11T05:12:56

From https://console.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic2.html#using_spark-submit you need to be using Spark version 1.6.x or 2.0.x. Your submit job is set to version 2.2.0. Try submitting using spark.service.spark_version=2.0.0 (assuming your code will work with this version of Spark).

spark-submit on ibm bluemix

1 Answers