Log Configuration for Kubernetes multiple POD instances for same services

Question

We have recently moved to micro services based architecture for our enterprise application. we are using Kubernetes cluster to host all our micro services.

Currently we didn't configure ELK for manage our logs, just storing application logs into azure blob storage.

We are facing issue, when multiple POD instances are running for one services, since all instances use same log file to update the content. due to this, instances are getting stuck and getting memory leak issue.

I have configured mount path in docker container , and my logback property , has below entry to write the logs.

<property name="DEV_HOME" value="/mnt/azure/<service-name>/logs" />

Is there a way to get the pod instance name in log configuration , so that i can add one more level down, to have separate logs for different instances.

Or is there better way to handle this scenario.

<property name="DEV_HOME" value="/mnt/azure/<service-name>/<instances>/logs" />

Answer 1

It should be possible to set the Pod information (including the name) as environment variables as mentioned here. In the application read the environment variable and log appropriately.

Answer 2

As mentioned by @PraveenSripati, downward api is the solution. However, there are cases where one is forced to use third-party libraries and not able to use environment variables to override the location. In past we have configured our pods to use a combination of downward API and a K8S command and args to run a custom script before starting the application.

Let's assume the log location is set to /opt/logs, in that case, we do something like below, We keep this script in the container while builing it. For instance we call this entrypoint.sh ,

#!/bin/bash
log_location="/opt/log-$MY_POD_NAME/$MY_POD_IP"
mkdir -p /opt/log-$MY_POD_NAME/$MY_POD_IP 
rm -f /opt/log 
ln -s /opt/log $log_location
exec <Application_start_command> 

Kubernetes POD definition

apiVersion: v1
kind: Pod
metadata:
  name: dapi-envars-fieldref
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "sh", "-c"]
      args:
      - /opt/entrypoint.sh 
      env:
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_SERVICE_ACCOUNT
          valueFrom:
            fieldRef:
              fieldPath: spec.serviceAccountName
  restartPolicy: Never

This will create a symlink where the third-party library can write without changing any code there.