add a new column to spark dataframe from array list

1
votes

I am trying to add a new column to spark data frame as below:

val abc :Array[String] = ("a","b","c","d")

I am trying to add this Array[String] as new column to dataframe and trying to do sha2 on that new column

var words=abc.mkString("||") source = source.withColumn("newcolumn", sha2(col(words), 256).cast(StringType))

It complied and the runtime error i am getting as

Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'a||b||c||d||e' given input columns:

The expected output should be a dataframe with newcolum as column name and the value as varchar64 with sha2 of concatenate of Array of string with ||.

Anyhelp is appreciated.

1
scalaapache-spark

1 Answers

0
votes

You need to use lit(words) and caluculate sha2(col,256)

Example

import org.apache.spark.sql.functions._

var words=abc.mkString("||")

val source = source.withColumn("newcolumn",sha2(lit(words),256))
source.show()
//+----------------------------------------------------------------+
//|newcolumn                                                       |
//+----------------------------------------------------------------+
//|b51ff0f0a6aa9f36848b8fdc10ece7242698a4061e141c761b1ac9a08634c067|
//+----------------------------------------------------------------+