Is it possible to selectively forward the log files to specific indexes in Splunk.
I want to forward a docker container running 3 services logs to Splunk indexer, the problem is that if I use Docker logging driver, all the data written to STDOUT goes to the same index and data segregation is not possible. Instead of that I've setup forwarder and able to send logs but all are going to the same index, I want to configure splunk forwarder to send specific logs to a specific index.
Let me start from the beginning. Running multiple processes in the same container is an anti-pattern. Try to avoid it as much as possible. Kubernetes, for example, have a great solution for your case, where they can deploy two containers in the same Pod and just setup communication between the containers on the same loopback network interface (127.0.0.1), so for the processes, it will look like they are running in the same container. See https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/ for details.
If you still want to have all three processes in the same container, you have two options to get the logs in different indices:
routing on indexer
If you can identify logs on side of indexer, you can forward logs as you usually do and using transforms.conf on the indexer side route them to specific index, see http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf
[nginx_route]
DEST_KEY = _MetaData:Index
REGEX = nginx .*
FORMAT = index_nginx
avoiding container logs
Another option. You can create a volume for logs, that you will share between your container and forwarder (Splunk Universal Forwarder) or our collector (https://www.outcoldsolutions.com). And in the configuration define to which index you want to forward these logs. In you container you will need to change how you write logs, instead of stdin, write them to the files.
