We're interested in forwarding the logs from a node.js server running in a Docker container to Splunk.
Some options we've considered include a side-car container running a Splunk forwarder. The side-car would write to a shared volume that the side-car would observe and send on.
Ideally, we would just use a syslog drain or another mechanism, but I can't seem to find any documentation on how to set that up?
There are a lot of options to send logs from containers to Splunk.
For logs, sent to Standard Output and Error:
For application logs (logs written inside of the container):
Our company (https://www.outcoldsolutions.com) offers one solution that can simply forward container (https://www.outcoldsolutions.com/docs/monitoring-docker/v5/) and application logs (https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#application-logs) from the Docker hosts, and collect metrics. We also provide you with an application in Splunk for tracking the health and performance of your clusters https://splunkbase.splunk.com/app/3723/. Our application is not free, but cheap compared to the time you can spend building something similar.
Another option is using fluentd as an intermediary.
Fluentd exists as docker logging driver as well, but you can use it to redirect the logs to several backends (Splunk, Elasticsearch). You are not as tightly coupled to Splunk.
Additionally that's the way proposed by Openshift.
