Would Spark unpersist the RDD itself when it realizes it won't be used anymore?

32
votes

We can persist an RDD into memory and/or disk when we want to use it more than once. However, do we have to unpersist it ourselves later on, or does Spark does some kind of garbage collection and unpersist the RDD when it is no longer needed? I notice that If I call unpersist function myself, I get slower performance.

2
apache-sparkhadooprdddistributed-computing
If you cache an RDD, you'll have to unpersist yourself! – eliasah
@eliasah what happens if the memory is full? Doesn't spark unpersist the RDD's in LRU fashion. – None
Nope it doesn't. Spark isn't a cache system. You might consider using and external cache, Or you want to persist on disk and on ram. Nevertheless, if there is no space on the disk, you'll get an not space available on device error. – eliasah
@eliasah: Interesting, my understanding is exactly the opposite of yours. 1) The RDD will be unpersisted when GCd. 2) Memory pressure will also push out the RDD from the cache. 3) A big part of Spark is a cache system. I hope you can post your references. I posted an answer regarding the unpersist behavior, so you can also correct me there if I'm wrong. Thanks! – Daniel Darabos
Haha, you're right — it's certainly not advertised as a "cache system". Also I'm not sure if it does LRU or FIFO or what. By the way I skimmed past your mention of disk earlier. There is a good point there: disk space on the executors (used by RDDs persisted to disk and shuffle files) is getting cleaned up in response to GC on the driver. There is a danger of the executors filling up the disk before a GC would be triggered on the driver. We call System.gc() at certain points to try to avoid this. – Daniel Darabos

2 Answers

27
votes

Yes, Apache Spark will unpersist the RDD when it's garbage collected.

In RDD.persist you can see:

sc.cleaner.foreach(_.registerRDDForCleanup(this))

This puts a WeakReference to the RDD in a ReferenceQueue leading to ContextCleaner.doCleanupRDD when the RDD is garbage collected. And there:

sc.unpersistRDD(rddId, blocking)

For more context see ContextCleaner in general and the commit that added it.

A few things to be aware of when relying on garbage collection for unperisting RDDs:

  • The RDDs use resources on the executors, and the garbage collection happens on the driver. The RDD will not be automatically unpersisted until there is enough memory pressure on the driver, no matter how full the disk/memory of the executors gets.
  • You cannot unpersist part of an RDD (some partitions/records). If you build one persisted RDD from another, both will have to fit entirely on the executors at the same time.
0
votes

As pointed out by @Daniel, Spark will remove partitions from the cache. This will happen once there is no more memory available, and will be done using a least-recently-used algorithm. It is not a smart system, as pointed out by @eliasah.

If you are not caching too many objects you don't have to worry about it. If you cache too many objects, the JVM collection times will become excessive, so it is a good idea to unpersist them in this case.