How to change time-to-live of GKE cluster events (from default 1h0m0s)?

Question

I am running GKE k8s deployments/jobs that take a long time to execute - from days to weeks (Machine Learning). Default GKE Events expire after 1h, which is not enough to debug problems that can occur during training, like OOMKilling, etc.

kube-apiserver isn't exposed in GKE, so I'd like to find a way to access and change a property like event-ttl.

How can I change event-ttl for an already launched cluster, or how to specify event-ttl at a cluster creation? For example if I would like all my events in a cluster to be available for 24 hours. Is there a kubectl command for that (the kubernetes svc is accessible)?

Thanks in advance!

Answer 1

It's currently not possible. As you said, the kube-apiserver flags are currently not user-configurable on GKE. There are plans to expose more such knobs to the users but I don’t know when would this be available.