spark dataframe map aggregation with alias?

Question

I like to use spark dataframe map aggregation syntax like this:

jaccardDf
        .groupBy($"userId")
        .agg(
          "jaccardDistance"->"avg"
          , "jaccardDistance"->"stddev_samp"
          , "jaccardDistance"->"skewness"
          , "jaccardDistance"->"kurtosis"
)

Is there a way to alias the resulting columns while still using the Map syntax? When I need to alias I do this instead

jaccardDf
        .groupBy($"userId")
        .agg(
          avg("jaccardDistance").alias("jaccardAvg")
          ,stddev_samp("jaccardDistance").alias("jaccardStddev")
          ,skewness("jaccardDistance").alias("jaccardSkewness")
          ,kurtosis("jaccardDistance").alias("jaccardKurtosis")
)

Are you sure the map sytanx result in 4 different columns? Shouldn't each new instance of "jaccardDistance" replace previous instance and result in single aggregation — D3V
Yes, I'm sure. I get the usual column names when calculating and not aliasing like "avg(jaccardDistance)" — Michael West

caffreyd caffreyd · Accepted Answer · 2019-12-06T20:54:34

Use .toDF() to alias your column names with a list you define:

val colNames = Array("userId", "jaccardAvg", "jaccardStddev", "jaccardSkewness", "jaccardKurtosis") 

jaccardDf
    .groupBy($"userId")
    .agg(
      "jaccardDistance"->"avg",
      "jaccardDistance"->"stddev_samp",
      "jaccardDistance"->"skewness",
      "jaccardDistance"->"kurtosis")
    .toDF(colNames: _*)

spark dataframe map aggregation with alias?

1 Answers