Writing SQL table directly to file in Scala

1
votes

Team, I'm working on Azure databricks, I'm able to write a dataframe to CSV file using the following option:

df2018JanAgg
.write.format("com.databricks.spark.csv")
.option("header", "true")
.save("dbfs:/FileStore/output/df2018janAgg.csv")

but I'm seeking an option to write data directly from SQL table to CSV in Scala. Can someone please let me know if such options exist.

Thanks, Srini

1
scalaexport-to-csvazure-databricks

1 Answers

0
votes

Yes data could be directly loaded between a sql table to Datafame and vice-versa. Reference: https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html

//JDBC -> DataFarme -> CSV
spark.read
.format("jdbc")
.option("url", "jdbc:postgresql:dbserver")
.option("dbtable", "schema.tablename")
.option("user", "username")
.option("password", "password")
.load()
.write.format("com.databricks.spark.csv")
.option("header", "true")
.save("dbfs:/FileStore/output/df2018janAgg.csv")

//DataFarme -> JDBC
df.write
.format("jdbc")
.option("url", "jdbc:postgresql:dbserver")
.option("dbtable", "schema.tablename")
.option("user", "username")
.option("password", "password")
.save()