Bamboo build-dir excessive space can it be cleaned up with a cron job?

10
votes

We use Bamboo CI. There are multiple bamboo local agents and parallel builds across many plans. The build-dir in bamboo-home is many hundreds of gigabytes, and analysis shows that it just continually grows as new feature branches are added. Plans seem to be duplicated in each local agent directory, and also directly in build-dir.

Unlike expiring artifacts, Bamboo does not seem to clean this up by itself. For example, if a local agent is removed then the local agents build directory sits there forever taking up a significant amount of space.

Plans can be set to clean up at the end of a build, however this impacts problem analysis in the event of needing to do a post-mortem on the build.

Due to the directory running out of space I have just added a daily cron task to periodically remove files and directories that haven't been accessed for more than 21 days. When I first ran this manually I reclaimed 300GB from a 600GB partition. I want to know if others have encountered this same issue, and if it is safe to externally clean the build-dir in the long term. Could it impact bamboo builds? Is there some bamboo option that I have missed that would do this for me?

Searching on the Atlassian site has not been helpful and yields no answers... what are others doing to tame this space hog?

2
continuous-integrationbamboodiskspace
I have similar setup and I delete all artifacts which are more than one month old, which frees huge chunk of space. I have written a script which runs at the end of every month. I don't think anyone uses data which is older than one week. - binarymemoir
I wasn't so much worried about artifacts, they do take up a lot of space but bamboo will clean them up itself based on the plan configuration and whether they are in the past 3 deployments to any environment. My question was more about the space consumed in the bamboo_home/xml-data/build-dir tree. - shonky linux user

2 Answers

17
votes

The cron job has been running for a while now without any issues, and it is keeping the space usage under control.

I have reduced the parameter to 15 days.

My crontab looks like this:

# clean up old files from working directory
0 20 * * * find /<path_to>/bamboo-home/xml-data/build-dir/ -depth -not -path *repositories-cache* -atime +15 -delete

# clean up old backups every Sunday
0 21 * * 0 find /<path_to>/bamboo-home/backups -type f -mtime +28 -delete

# remove any old logs from install directory after 15 days
0 22 * * * find /<path_to>/bamboo/logs/ -type f -mtime +15 -delete

# quick and dirty truncate catalina.out to stop it growing too large (or better still use logrotate) 
0 23 * * * cat /dev/null > /<path_to>/bamboo/logs/catalina.out

I hope this is useful for others trying to tame bamboo's diskspace usage. The first job is the important one, the last three are just housekeeping.

N.B. logrotate is not used on catalina.out due to unique circumstances in my companies outsourced linux environment. I would generally recommend logrotate if possible rather than my quick and dirty truncate method - see answer by Jon V.

3
votes

While the cron idea works well - the thing that I've also done in the past with Bamboo is to "Clean working directory after each build" options. Basically, for any given job, there's a config option that will clean up the appropriate build-dir/<build_plan_job> directory for a given plan/job:

Actions -> Configure Plan -> click the Job -> Miscelaneous Tab -> first checkbox

While that makes sure that future build scratch areas are cleaned up, it does not help for already existing and/or old builds. Given the normal git style workflow where you have lots of branches (and each branch creates a specific job ID for it (like PLAN-JOB_WITH_BRANCH_NUMBER-BUILD_NUMBER or similar) that gets old/large fast. I just did a quick check, and we're now cleaning up the build areas for most builds (the large ones at least), but we have over 100Gig of build cruft from branches that have been merged loooong ago.

Thanks for the cron example, though, that should work OK for the future.

Unrelated: the more I use Bamboo, the more I love/hate it.

EDIT: as a general comment, I'd try really hard to work with an SA to get a logrotate rule set up/implemented for the catalina.out - overwriting with /dev/null seems like a really bad idea, unless you're already slurping them up with something like ELK or Splunk.

My /etc/logrotate.d/bamboo_catalina_out looks like (using your paths):

/<path_to>/bamboo/logs/catalina.out {
  create 0660 bamboo bamboo
  compress
  copytruncate
  missingok
  rotate 10
  size 100M
}

Finally - is there a reason why you have both the third and fourth cron scripts?