Spark iterative/recursive algorithms - Breaking spark lineage

0
votes

I have a recursive spark algorithm that applies a sliding window of 10 days to a Dataset.

The original dataset is loaded from a Hive table partitioned by date.

At each iteration a complex set of operations is applied to Dataset containing the ten day window.

The last date is then inserted back into the original Hive table and the next date loaded from Hive and unioned to the remaining nine days.

I realise that I need to break the spark lineage to prevent the DAG from growing unmanageable.

I believe I have two options:

  1. Checkpointing - involves a costly write to HDFS.

  2. Convert to rdd and back again

    spark.createDataset(myDS.rdd)

Are there any disadvantages using the second option - I am assuming this is an in memory operation and is therefore cheaper.

1
apache-sparkapache-spark-sql

1 Answers

4
votes

Check pointing and converting back to RDD are indeed the best/only ways to truncate lineage.

Many (all?) of the Spark ML Dataset/DataFrame algorithms are actually implemented using RDDs, but the APIs exposed are DS/DF due to the optimizer not being parallelized and lineage size from iterative/recursive implementations.

There is a cost to converting to and from RDD, but smaller than the file system checkpointing option.