Java Spark alternative for groupBy

Question

I have a dataset with column1 and column2. I want to assign new column value "x" to the records if "column1" value occurs more than once in the dataset. I can use group by to achieve this.

dataset.groupBy("column1").count()

But this method creates a group by dataset and then I have to do a join between the group by dataset and original dataset to find the count and assign new value as "X" if count is more than 1. Is it possible to achieve the same result without creating a new dataset ?

Answer 1

You can use a window function to add a column of count:

dataset.withColumn("count_column2", count("column2").over(Window.partitionBy("column1")))