Where do docker logs go when splunk server is not reachable

Question

I am trying to get docker logs from a running docker container. I have configured splunk as logging driver in my docker compose and I understand that if splunk server is not reachable then container won't start.

  proxysecurity:
    image: test/image
    network_mode: host
    depends_on:
      - zookeeper
    ports:
      - '8083:8083'
    logging:
      driver: "splunk"
      options:
         splunk-url: "http://XX.X2.X3.X1:XXX7/"
         splunk-token: "XXXXX5-9CA1-44B8-B9E8-2XXX25"
         splunk-format: json
         tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
    environment:
      XXXCONNECT: localhost:32181
      XXXXXRS: http://localhost:8083

Now if splunk server is not reachable when container is up and running , is there any fall back mechanism wherein we can tell docker container to log locally ? Or is there any way to log to splunk as well as locally inside container ?

outcoldman outcoldman · Accepted Answer · 2017-12-06T15:56:59

I am the author of the Splunk Logging Driver.

In case if Splunk is unavailable, driver holds small buffer in memory and keeps retrying. Configuration for the size of the buffer is documented no official docs for the driver https://docs.docker.com/engine/admin/logging/splunk/

SPLUNK_LOGGING_DRIVER_BUFFER_MAX If driver cannot connect to remote server, what is the maximum amount of messages it can hold in buffer for retries.

Unfortunately this is not ideal, considering that this buffer can be filled pretty quickly and that increasing the buffer to higher number can affect your containers. But this is how most of the drivers written.

I have built another solution delivering logs and metrics to Splunk. This solution includes tiny image with collector and Splunk Certified Application. It is built on top of json-file driver, which means that when Splunk is unavailable it will just keep retrying from the position of the log files. The logs files can have their own settings for rotation, this can be configured with dockerd daemon configuration. You can read another benefits of our solution Comparing with Splunk Logging Driver. And how to get started with Monitoring Docker.

Where do docker logs go when splunk server is not reachable

1 Answers