Split splunk log-driver for Docker into multiple indexes

Question

I have a small java application running inside of docker. The application is a Dropwizard application, meaning it has java classes that log with log4j, and web endpoints, (JaxRS) that has request logging capability.

Now, I want to forward my container logs to Splunk. The issue is I really want them split into two indexes; one for the application log, and one for the request log. We can call the indexes cool_app and cool_req.

Is there any way of doing this? Starting my dropwizard application causes it to output both the application log and request log to the stdout. (Whilest also logging to two files)

I've tried the docker logdriver but it seems like it's only made for one index.

I might add that it's easy to separate the logs by regEx if that would be an option

Answer 1

Let me start first by suggesting (if that is possible) to split this container in multiple containers. Running multiple processes in the same container is an anti-pattern and should be avoided. So this is basically a reason why you have this question. If you will use Kubernetes - it will allow you to schedule multiple containers in the same Pod, allowing them to share the same volumes and loopback interface, giving you a lot of various options for inter-process communication.

But I understand that it is not always possible to do. In that way, possibly the best solution will be to keep these logs as application logs, meaning that you will have a data volume for this container, that will store these logs, and not going to forward them to standard output. To get these logs to Splunk you can use Sidecar container with Splunk Universal Forwarder or our collector (if you are our customer or wants to be).

If side container is not an option, the last choice to keep the logs in the container and add a Splunk UF to the image and run it together with the application process. https://docs.docker.com/config/containers/multi-service_container/

Btw, we are working on some new features, that will allow forwarding application logs with the minimum configuration changes.