Filebeat with ELK stack running in Kubernetes does not capture pod name in logs

4
votes

I am using the ELK stack (elasticsearch, logsash, kibana) for log processing and analysis in a Kubernetes (minikube) environment. To capture logs I am using filebeat. Logs are propagated successfully from filebeat through to elasticsearch and are viewable in Kibana.

My problem is that I am unable to get the pod name of the actual pod issuing log records. Rather I only get the filebeat podname which is gathering log files and not name of the pod that is originating log records.

The information I can get from filebeat are (as viewed in Kibana)

  • beat.hostname: the value of this field is the filebeat pod name
  • beat.name: value is the filebeat pod name
  • host: value is the filebeat pod name

I can also see/discern container information in Kibana which flow through from filebeat / logstash / elasticsearch:

  • app: value is {log-container-id}-json.log
  • source: value is /hostfs/var/lib/docker/containers/{log-container-id}-json.log

As shown above, I seem to be able to get the container Id but not the pod name.

To mitigate the situation, I could probably embed the pod-name in the actual log message and parse it from there, but I am hoping there is a solution in which I can configure filebeat to emit actual pod names.

Does anyone now how to configure filebeat (or other components) to capture kubernetes (minikube) pod names in their logs?

My current filebeat configuration is listed below:

ConfigMap is shown below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat
  namespace: logging
  labels:
    component: filebeat
data:
  filebeat.yml: |
    filebeat.prospectors:

    - input_type: log
      tags:
      - host
      paths:
      - "/hostfs/var/log"
      - "/hostfs/var/log/*"
      - "/hostfs/var/log/*/*"
      exclude_files:
      - '\.[0-9]$'
      - '\.[0-9]\.gz$'

    - input_type: log
      tags:
      - docker
      paths:
      - /hostfs/var/lib/docker/containers/*/*-json.log
      json:
        keys_under_root: false
        message_key: log
        add_error_key: true
      multiline:
        pattern: '^[[:space:]]+|^Caused by:'
        negate: false
        match: after

    output.logstash:
      hosts: ["logstash:5044"]

    logging.level: info

DamemonSet is shown below:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  template:
    metadata:
      labels:
        component: filebeat
    spec:
      containers:
      - name: filebeat
        image: giantswarm/filebeat:5.2.2
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat
          readOnly: true
        - name: hostfs-var-lib-docker-containers
          mountPath: /hostfs/var/lib/docker/containers
          readOnly: true
        - name: hostfs-var-log
          mountPath: /hostfs/var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat
      - name: hostfs-var-log
        hostPath:
          path: /var/log
      - name: hostfs-var-lib-docker-containers
        hostPath:
      path: /var/lib/docker/containers
3
elasticsearchlogginglogstashkubernetesfilebeat

3 Answers

1
votes

disclaimer: I'm a beats developer

What you want to do is not yet supported by filebeat, but definitely, it's something we want to put some effort on, so you can expect future releases supporting this kind of mapping.

In the meantime, I think your approach is correct. You can append the info you need to your logs so you have it in elasticsearch

1
votes

I have achieved what you looking for, by assigning a group of specific pods to a namespace, and now can query the log I look for using a combination of namespace, pod name and container name which is also included in generated log which is piped by file beat without any extra effort as you can see here image

1
votes

For future people coming here, it is now already in place in a filebeat processor :

filebeat.prospectors:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
  - type: docker
    containers.ids:
    - "*"
    processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_event:
          when:
            equals:
              kubernetes.container.name: "filebeat"

helm chart default values : https://github.com/helm/charts/blob/master/stable/filebeat/values.yaml

doc : https://www.elastic.co/guide/en/beats/filebeat/current/add-kubernetes-metadata.html