Parsing Kubernetes syslog logs with Logstash

0
votes

Kubernetes leaves logs in /var/log/syslog. Example:

I1021 12:57:47.052671   35084 setters.go:73] Using node IP: "10.11.22.33."

I understand that the I stands for info, followed by a certain code, then a timestamp. But what is the code after that? What does setters.go mean? I also see that the actual log message comes after ].

I would like to parse these with Logstash, but I couldn't find any documentation about what each field in the header supposed to mean.

I could write these parsers myself with the grok filter, of course, but first I would like to understand what does each message mean.

1
loggingkuberneteslogstashkubectllogstash-grok

1 Answers

0
votes

I1021 12:57:47.052671 35084 setters.go:73] Using node IP: "10.11.22.33."

/var/log/syslog or /var/log/messages: general messages, as well as system-related information. Essentially, this log stores all activity data across the global system. Note that activity for Redhat-based systems, such as CentOS or Rhel, are stored in messages, while Ubuntu and other Debian-based systems are stored in Syslog.

This is kubelet entry:

setters.go - is script written in GO that updates the daemon endpoints on the node.

73 - is number of line which prints the message

klog.V(2).Infof("Using node IP: %q", nodeIP.String())

If you are using systemd you should be using jurnalctl -u kubelet

You should also check Kubernetes documentation for Looking at logs.

For now, digging deeper into the cluster requires logging into the relevant machines. Here are the locations of the relevant log files. (note that on systemd-based systems, you may need to use journalctl instead)

Master

  • /var/log/kube-apiserver.log - API Server, responsible for serving the API
  • /var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions
  • /var/log/kube-controller-manager.log - Controller that manages replication controllers

Worker Nodes

  • /var/log/kubelet.log - Kubelet, responsible for running containers on the node
  • /var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing

If you are looking for Application logs then please check Troubleshoot Applications.