Not able to apply function to Spark Dataframe Column

Question

I am trying to apply a function to one of my dataframe columns to convert the values. The values in the column are like "20160907" I need value to be "2016-09-07".

I wrote a function like this:

def convertDate(inDate:String ): String = {
   val year = inDate.substring(0,4)
   val month = inDate.substring(4,6)
   val day = inDate.substring(6,8)

   return year+'-'+month+'-'+day
}

And in my spark scala code, I am using this:

def final_Val {
  val oneDF = hiveContext.read.orc("/tmp/new_file.txt")
  val convertToDate_udf = udf(convertToDate _)
  val convertedDf = oneDF.withColumn("modifiedDate", convertToDate_udf(col("EXP_DATE")))
  convertedDf.show()
}

Suprisingly, in spark shell I am able to run without any error. In scala IDE I am getting the below compilation error:

Multiple markers at this line:
not enough arguments for method udf: (implicit evidence$2: 
reflect.runtime.universe.TypeTag[String], implicit evidence$3: reflect.runtime.universe.TypeTag[String])org.apache.spark.sql.UserDefinedFunction. Unspecified value parameters evidence$2, evidence$3.

I am using Spark 1.6.2, Scala 2.10.5

Can someone please tell me what I am doing wrong here?

Same code I tried with different functions like in this post: stackoverflow.com/questions/35227568/applying-function-to-spark-dataframe-column". I am not getting any compilation issues with this code. I am unable to find out the issue with my code

Here is a similar answer you can change the dateformat as you want https://stackguides.com/questions/44018393/change-column-value-in-a-dataframe-spark-scala/44020820#44020820 — koiralo
@RobertHorvick Because udf needs a function as input, not a method. ith the underscore you "lift" the method to a function — Raphael Roth

Haroun Mohammedi Haroun Mohammedi · Accepted Answer · 2017-05-18T18:35:10

From what I have learned in a spark-summit course, you have to use the sql.functions methods as much as possible. before implementing your own udf you have to check if there's no existing function in the sql.functions package that does the same work. using the existing functions spark can do a lot of optimizations for you and it will not be obliged to serialize and deserialize you data from and to JVM objects.

to achieve the result you want I'm gonna propose this solution :

val oneDF = spark.sparkContext.parallelize(Seq("19931001", "19931001")).toDF("EXP_DATE")
val convertedDF = oneDF.withColumn("modifiedDate", from_unixtime(unix_timestamp($"EXP_DATE", "yyyyMMdd"), "yyyy-MM-dd"))
convertedDF.show()

this gives the following results :

+--------+------------+
|EXP_DATE|modifiedDate|
+--------+------------+
|19931001|  1993-10-01|
|19931001|  1993-10-01|
+--------+------------+

Hope this help. Best Regards

Not able to apply function to Spark Dataframe Column

1 Answers