How can Splunk distinguish between output to stdout and stderr in Docker context?

Question

We have a Java application that can be run in Docker containers. It produces messages to stdout and stderr with a different level of detail for different audiences.

Configuring Splunk as log driver all log lines received by Splunk a marked with source stdout although there must be log lines being logged to stderr.

Splunk log driver configuration in docker-compose:

logging:
    driver: splunk
    options:
        splunk-url: https://splunkhf:8088
        splunk-token: [TOKEN]
        splunk-index: splunk_index
        splunk-insecureskipverify: "true"
        splunk-sourcetype: log4j
        splunk-format: "json"
        tag: "{{.Name}}/{{.ID}}"

Example log message sent to splunk:

{
   line: 2021-01-12 11:37:49,191;10718;INFO ;[Thread-1];Logger; ;Executed all shutdown events. 
   source: stdout 
   tag: service_95f2bac29286/582385192fde 
}

How can I configure Docker or Splunk to differentiate correctly between those different streams?`

Markus Markus · Accepted Answer · 2021-01-21T06:15:53

If you run the service from docker-compose without -d then the logs lose their original source. It seems that Docker and Docker-Compose put everything from the container's output streams to stdout and use stderr for their logs.

Using the -d flag the log messages do not lose their original output stream.

How can Splunk distinguish between output to stdout and stderr in Docker context?

1 Answers