local cache post map reduce jobs taking space

Question

I am using Hortonworks environment for big data processing.

I observed that as part of Map Reduce jobs from Hive/Oozie/Pig etc my below local cache directory is getting filled up.It creates one folder for every map reduce execution with the jar files.

/hadoop/yarn/local/usercache/root/filecache

Is it fine if I delete these folders directly? is there any configuration that I can set to remove it automatically?

franklinsijo franklinsijo · Accepted Answer · 2017-02-13T10:14:42

These directories should be cleaned up automatically every 600000ms (10 minutes).

Or you can reduce this timeperiod by setting up this property in yarn-site.xml, yarn.nodemanager.localizer.cache.cleanup.interval-ms.

You can also change the maximum size this cache directory should occupy, yarn.nodemanager.localizer.cache.target-size-mb. The default target size is 10240MB.

If no jobs are running, manual deletion of cache is fine. Restart nodemanager after deleteing the cache.

local cache post map reduce jobs taking space

1 Answers