How to cast a WrappedArray[WrappedArray[Float]] to Array[Array[Float]] in spark (scala)

8
votes

Im using Spark 2.0. I have a column of my dataframe containing a WrappedArray of WrappedArrays of Float.

An example of a row would be:

[[1.0 2.0 2.0][6.0 5.0 2.0][4.0 2.0 3.0]]

Im trying to transform this column into an Array[Array[Float]].

What I tried so far is the following:

dataframe.select("mycolumn").rdd.map(r => r.asInstanceOf[Array[Array[Float]]])

but I get the following error:

Caused by: java.lang.ClassCastException:
 org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to [[F

Any idea would be highly appreciated. Thanks

2
arraysscalacastingspark-dataframeapache-spark-2.0

2 Answers

4
votes

Try this:

  val wawa: WrappedArray[WrappedArray[Float]] = null 
  val res: Array[Array[Float]] = wawa.map(inner => inner.array).toArray

It compiles for me

3
votes

Following @sami-badawi 's answer I am posting the answer for those like me who started from a dataframe.

dataframe.select("mycolumn").rdd.map
(row => row.get(0).asInstanceOf[WrappedArray[WrappedArray[Float]]].array.map(x=>x.toArray))