Spark Dataframe to TXT file without carriage return

Question

I am trying to save the spark dataframe as text file. While doing this, I need to have specific column delimiter and row delimiters. I am unable to get the row delimiter working. Any help would be greatly appreciated. Below is the sample code for reference.

//option -1

spark.sparkContext.hadoopConfiguration.set("textinputformat.record.delimiter", "\\§")
df.coalesce(1)
   .map(_.mkString("\u00B6"))
   .write
   .option("encoding", "US-ASCI")
   .mode(SaveMode.Overwrite).text(FileName)

//option-2

      df.coalesce(1)
      .write.mode(SaveMode.Overwrite)
      .format("com.databricks.spark.csv")
        .option("inferSchema", "true")
      .option("encoding", "US-ASCI")
        .option("multiLine", false)
      .option("delimiter", "\u00B6")
        .option("lineSep", "\u00A7")
      .csv(FileName1)

Below is my input and output for reference:

Input:

Test1,Test2,Test2
Pqr,Rsu,Lmn
one,two,three

Output:

Test1¶Test2¶Test2§Pqr¶Rsu¶Lmn§one¶two¶three

Chris H. Chris H. · Accepted Answer · 2019-09-24T22:13:41

From Spark 2.4.0, the "lineSep" option can be used to write json and text files with a custom line separator (cf. DataFrameWriter spec). This option is ignored in previous Spark versions and for csv format.

val df = spark.createDataFrame(Seq(("Test1","Test2","Test2"), ("one","two","three")))

df.map(_.mkString("\u00B6"))
  .coalesce(1)
  .write
  .option("lineSep", "\u00A7")
  .text(FileName)

Output with Spark 2.4.*:

Test1¶Test2¶Test2§one¶two¶three

Output with Spark 2.3.* and lower (the "lineSep" option is ignored):

Test1¶Test2¶Test2
one¶two¶three

Spark Dataframe to TXT file without carriage return

1 Answers