AKS log format changed

2
votes

we recently updated our AKS cluster from 1.17.x to 1.19.x and recognised that the format of our custom application logs in /var/lib/docker/containers changed.

Before the update it looked like this: old valid json format

Afterwards it looks like this: new invalid json format

I can find some notes in the changelog that kubernetes changed from just text logs to structured logs (for system components) but I don't see how this correlates to how our log format changed.

https://kubernetes.io/blog/2020/09/04/kubernetes-1-19-introducing-structured-logs/#:~:text=In%20Kubernetes%201.19%2C%20we%20are,migrated%20to%20the%20structured%20format

https://kubernetes.io/docs/concepts/cluster-administration/system-logs/

Is there a chance to still get valid json logs to /var/lib/docker/containers in AKS > 1.19.x?

Background: We send our application logs to Splunk and don't use the Azure stack for log analysis. Our Splunk setup cannot parse that new log format as of now.

1
azurekubernetesazure-akssplunk

1 Answers

4
votes

The format of the logs are defined by the container runtime. It seems before you were parsing logs from docker container runtime, and now it is containerd (https://azure.microsoft.com/en-us/updates/azure-kubernetes-service-aks-support-for-containerd-runtime-is-in-preview/).

Based on the article - you can still choose moby (which is docker) as the container runtime.

To take that also from your shoulders, you should look into using one of those (considering that they will automatically detect the log format and container runtime for you).