How to clean-up old unused Kubernetes images/tags?

Question

To simplify deployment and short term roll-back, it's useful to use a new Docker image tag for each new version to deploy on Kubernetes. Without clean-up this means that old images:tags are kept forever.

How can I list all image:tag that are used by a Kubernetes container so that I can find all old image:tag that are old and not used to delete them automatically from the Docker Registry?

My goal is ideally for Google Container Engine (GKE) to delete unused images a Google Container Registry.

Answer 1

As an alternative approach, you might consider just letting Kubernetes handle reclamation of old images for you.

Presently, the ImageManager handles reclamation of candidate images. See: Garbage Collection

Garbage collection is a helpful function of kubelet that will clean up unreferenced images and unused containers. kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.

Configuration is controlled via these two kublet cli parameters:

  --image-gc-high-threshold=90: The percent of disk usage after which image garbage collection is always run. Default: 90%
  --image-gc-low-threshold=80: The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Default: 80%

The high/low thresholds could be tuned to force collection at an interval that works for you.

Answer 2

With recent versions of kubelet use the below options as --image-gc-high-threshold and --image-gc-low-threshold are being deprecated:

--eviction-hard
--eviction-soft

More details avaialble here:

• https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/
• https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

Answer 3

You could use docker-cleanup containers running in a DaemonSet. That would cleanup any unused images on each node in your cluster.

Answer 4

To get a list of all images used by a Kubernetes cluster, one can run the shell script:

for image in $(kubectl get pods --all-namespaces --output=jsonpath='{..image}')
do
    echo $image
done

It seems however that there is no way to simply currently delete images from a Google Container Registry (see How to remove a pushed image in Google Container Registry)

Answer 5

I am not sure if there is a documented approach to do that kind of maintenance. However Openshift Origin does attempt to tackle by pruning docker images and interacting with registry to remove older blobs

We have implemented it in context to origin. The source code for that it on github