How to convert RDD[some case class] to csv file using scala?

Question

I have an RDD[some case class] and I want to convert it to a csv file. I am using spark 1.6 and scala 2.10.5 .

stationDetails.toDF.coalesce(1).write.format("com.databricks.spark.csv").save("data/myData.csv")

gives error

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org
    at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
    at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:219)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139)

I am not able to add the dependencies for "com.databricks.spark.csv" in my build.sbt file.

dependencies I added in build.sbt file are:

libraryDependencies ++= Seq(
  "org.apache.commons" % "commons-csv" % "1.1",
  "com.univocity" % "univocity-parsers" % "1.5.1",
  "org.slf4j" % "slf4j-api" % "1.7.5" % "provided",
  "org.scalatest" %% "scalatest" % "2.2.1" % "test",
  "com.novocode" % "junit-interface" % "0.9" % "test"
)

I also tried this

stationDetails.toDF.coalesce(1).write.csv("data/myData.csv")

but it gives error : csv cannot be resolved.

I think https://github.com/databricks/spark-csv is what you want. — Jiayi Liao
I have already went through the link. I think I am not adding the dependencies in correct way.Apart from how I have added the dependencies mentioned above, is there anything else to make it work? @JiayiLiao — siddhau jain
@siddhaujain: In the dependencies above you haven't listed/added the library linked by Jiayi Liao, i.e.: "com.databricks" %% "spark-csv" % "1.5.0". — Shaido

a9207 a9207 · Accepted Answer · 2019-06-29T12:02:16

Please change your build.sbt to below -

libraryDependencies ++= Seq(
  "org.apache.commons" % "commons-csv" % "1.1",
  "com.databricks" %% "spark-csv" % "1.4.0",
  "com.univocity" % "univocity-parsers" % "1.5.1",
  "org.slf4j" % "slf4j-api" % "1.7.5" % "provided",
  "org.scalatest" %% "scalatest" % "2.2.1" % "test",
  "com.novocode" % "junit-interface" % "0.9" % "test"
)

How to convert RDD[some case class] to csv file using scala?

1 Answers