Changing columns that are string in Spark GraphFrame

Question

I'm using GraphFrame in spark 2.0 and scala.

I need to remove double quote from columns that are in string type (out of many columns). I'm trying to do so using UDF as follow:

import org.apache.spark.sql.functions.udf

val removeDoubleQuotes = udf( (x:Any) =>
    x match{
      case s:String => s.replace("\"","")
      case other => other
    }
  )

And I get the following error since type Any is not supported in GraphFrame.

java.lang.UnsupportedOperationException: Schema for type Any is not supported

What is a workaround for that?

Do your columns have mixed types? Why not just write it only for strings and apply it only to the string columns? — Joe K
@JoeK Because I have many columns and try to find a way rather than manually find string columns. — MehrdadAP

koiralo koiralo · Accepted Answer · 2017-07-08T05:24:58

I think you don't have a column with type Any and you can't return different datatype from udf. You need to have a single datatype return from udf.

If your column is String then you can create udf as

import org.apache.spark.sql.functions.udf

val removeDoubleQuotes = udf( (x:String) => s.replace("\"",""))

Changing columns that are string in Spark GraphFrame

1 Answers