Docker cleaning up Kubernetes Pod logs too quickly

1
votes

I am running a number of CronJobs to automate tasks on my home server. When a Job runs and its Pods complete, I can view logs using kubectl logs <podName>, however after a certain amount of time (less than 5mins but more than 30s) the Pod still exists in Kubernetes but I get:

unable to retrieve container logs for docker://<containerID>

I have not enabled any log rotation myself so I assume Docker is cleaning up the logs once the container finishes. How can I configure Docker to retain these container logs so that I can see the Jobs logs after a longer period of time (say 1 day) after the Job completes.

  • Cluster provisioner: kubeadm 1.17.0
  • Kubernetes version: 1.17.0
  • Docker version: 18.06.3-ce
  • uname -r: 4.19.86-coreos
2
dockerkubernetes

2 Answers

1
votes

You can try setting up minimum-container-ttl-duration flag in Kubelet. This is the minimum age for a finished container before it is garbage collected.

Also in the job spec you can specify a TTL

apiVersion: batch/v1
kind: Job
metadata:
  name: pi-with-ttl
spec:
  ttlSecondsAfterFinished: 100

The Job pi-with-ttl will be eligible to be automatically deleted, 100 seconds after it finishes.

0
votes

Went for a simple CronJob using docker prune

docker system prune -f -a --filter "until=168h"
docker volume prune -f