How to sort a dataset in Apache Flink?

7
votes

I have a Tuple Dataset of the form DataSet>. I wish to sort the "entire" Dataset on field String and then get only the Long values in a file. Flink does provide sort-partition but that does not help here as I need to sort the Dataset completely.

1
apache-flink
A sample of the data before and after? Maybe the code you have tried so far and how it failed? - Joshua Drake

1 Answers

13
votes

You can also use sortPartition() to sort the complete DataSet if you set the parallelism to 1:

DataSet<Tuple2<String, Long>> data = ...
DataSet<Tuple2<String, Long>> sorted = data
  .sortPartition(0, Order.ASCENDING).setParallelism(1); // sort in one partition
DataSet<Long> longs = sorted.map(new LongExtractor());  // map to extract long