How to display grouped data in Scala Dataframe

Question

I am a beginner in Scala, and I have a dataframe that looks like this (abbreviated):

root
 |-- contigName: string (nullable = true)
 |-- start: long (nullable = true)
 |-- end: long (nullable = true)
 |-- names: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- referenceAllele: string (nullable = true)
 |-- alternateAlleles: array (nullable = true)
 |    |-- element: string (containsNull = true)

I am trying to simply groupBy the names column:

display(dataframe.groupBy("names"))

a very simple operation, but

notebook:1: error: overloaded method value display with alternatives:
  [A](data: Seq[A])(implicit evidence$1: reflect.runtime.universe.TypeTag[A])Unit <and>
  (dataset: org.apache.spark.sql.Dataset[_],streamName: String,trigger: org.apache.spark.sql.streaming.Trigger,checkpointLocation: String)Unit <and>
  (model: org.apache.spark.ml.classification.DecisionTreeClassificationModel)Unit <and>
  (model: org.apache.spark.ml.regression.DecisionTreeRegressionModel)Unit <and>
  (model: org.apache.spark.ml.clustering.KMeansModel)Unit <and>
  (model: org.apache.spark.mllib.clustering.KMeansModel)Unit <and>
  (documentable: com.databricks.dbutils_v1.WithHelpMethods)Unit
 cannot be applied to (org.apache.spark.sql.RelationalGroupedDataset)
display(dataframe.groupBy("names"))

how can I display this grouped data?

Some of the solutions I've seen posted have been very complex, I don't think that this is a duplicate, what I want is extremely simple.

first, you cannot group over an array column, second you need to add an aggregation to create a DataFrame from the RelationalGroupedDataset — Raphael Roth

Aleh Pranovich Aleh Pranovich · Accepted Answer · 2019-09-18T19:16:32

groupBy returns RelationalGroupedDataset. You need to add any aggregation function (e.g. count()) dataframe.groupBy("names").count() or dataframe.groupBy("names").agg(max("end"))

If you need to group by each name, you can explode the "names" array before groupBy

dataframe
    .withColumn("name", explode(col("names"))) 
    .drop("names")
    .groupBy("name")
    .count()    // or other aggregate functions inside agg()

How to display grouped data in Scala Dataframe

1 Answers