Spark-1.5.0 - Loading com.databricks:-csv_2.11:1.2.0 in RStudio

Question

Having Spark-1.5.0 installed on my Mac machine, I'm trying to initialise spark context with com.databricks:-csv_2.11:1.2.0 package in rStudio, as:

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.databricks:-csv_2.11:1.2.0" "sparkr-shell"')
library(SparkR, lib.loc = "spark-1.5.0-bin-hadoop2.6/R/lib/")
sc <- sparkR.init(sparkHome = "spark-1.5.0-bin-hadoop2.6/")

But I'm getting the following error message:

[unresolved dependency: com.springml#spark-salesforce_2.10;1.0.1: not found]

Why does that happen?

P.s., the initiliztation works fine when I use com.databricks:spark-csv_2.10:1.0.3.

UPDATE

I tried to use the version com.databricks:spark-csv_2.10:1.2.0 and things work fine.

Now, I use this code in rStudio to load a csv file:

sqlContext <- sparkRSQL.init(sc)
flights <- read.df(sqlContext, "R/nycflights13.csv", "com.databricks.spark.csv", header="true")

I get the following error message:

Error in writeJobj(con, object) : invalid jobj 1

When I execute sqlContext I get the error:

Error in callJMethod(x, "getClass") : 
  Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed.

Session info:

R version 3.2.0 (2015-04-16)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.2 (Yosemite)

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] SparkR_1.5.0 rJava_0.9-7 

loaded via a namespace (and not attached):
[1] tools_3.2.0

Note that I don't get this error when I use Spark Shell with the same commands.

See my answer here: stackoverflow.com/q/32873434/1560062, 2.11 is Scala version and bre-built binaries are using Scala 2.10. If you want to use 2.11 you have to build Spark from source with Scala 2.11. Moreover you have typo in the package name - it should be com.databricks:spark-csv_2.11:1.2.0. If you simply want to use a recent version of spark-csv on 2.10 use com.databricks:spark-csv_2.10:1.2.0. — zero323
Thanks @zero323. Just tried to use com.databricks:spark-csv_2.10:1.2.0 , but I had the following error when I uploaded a csv file: Error in writeJobj(con, object) : invalid jobj 1 — Stan
The error "Error in writeJobj(con, object) : invalid jobj 1" does not occur when I use Spark shell instead of rStudio? any idea why? — Stan
Not really. I cannot reproduce this problem. Could you provide more details (OS, R version, RStudio version, session info)? — zero323

Stan Stan · Accepted Answer · 2015-10-01T16:27:47

Problem sovled.

Everything is working now after restarting the Rsession and using the following code:

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.databricks:spark-csv_2.10:1.2.0" "sparkr-shell"')
library(rJava)
library(SparkR, lib.loc = "spark-1.5.0-bin-hadoop2.6/R/lib/")

sc <- sparkR.init(master = "local", sparkHome = "spark-1.5.0-bin-hadoop2.6")

sqlContext <- sparkRSQL.init(sc)

flights <- read.df(sqlContext, "R/nycflights13.csv", "com.databricks.spark.csv", header="true")

Spark-1.5.0 - Loading com.databricks:-csv_2.11:1.2.0 in RStudio

1 Answers