Copy Data to Spark Environment

Question

I am trying to copy my data into spark and getting the following error

Error in as.data.frame.default(x, stringsAsFactors = FALSE, row.names = FALSE, : cannot coerce class "structure("SparkDataFrame", package = "SparkR")" to a data.frame

I've looked over the documentation and found nothing that helps

library(SparkR)
DF <- read.df("/FileStore/tables/export.csv", source = "csv", header="true", inferSchema = "true")

library(sparklyr)
library(dplyr)
sc <- spark_connect(method = "databricks")
#This gives the above error
DF_tbl <- sdf_copy_to(sc = sc, x = DF, overwrite = T)

wibeasley wibeasley · Accepted Answer · 2019-04-29T18:21:52

I can't test this without access to export.csv, but the problem may arise from mixing a SparkR function (ie, read.df()) with sparklyr functions (ie, spark_connect() and sdf_copy_to()).

I'm guessing that sparklyr::sdf_copy_to() doesn't accept an object of SparkR's 'SparkDataFrame' class.

Try replacing SparkR::read.df() with something like readr::read_csv() or data.table::fread().

Copy Data to Spark Environment

1 Answers