Journalctl : add _SYSTEMD_UNIT field into log printout

Question

Using the command:

/usr/bin/journalctl -o short -f | ncat {some-ip} {some port}

To forward journal output to some remote log tracking app.

Problem is that I'm missing the systemd unit / service name in the printout making it hard to tell which service produces what log line.

for example this is a nginx line :

Jun 25 07:51:09 localhost bash[497]: 10.23.132.98 - - [25/Jun/2014:07:51:09 +0000] "GET /page.html HTTP/1.1" 200 321 "https://{ip}" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"

In the log there is bash[497] - the PID of the process. How can I add more data to the log ? For example the docker container name of this PID or the systemd service/unit name ?

Answer 1

Don't use -o short, it removes information!

try -o json or -o verbose

Answer 2

Python can do this:

from systemd import journal

j = journal.Reader()
j.this_boot()
j.add_match(_SYSTEMD_UNIT="newnginx.service")
for entry in j:
   print('{} {}'.format(entry['_SYSTEMD_UNIT'], entry['MESSAGE']))

(python on CentOS 7)

Answer 3

In the end ive implemented this in a diffrent way

each service / systemd unit has a post execute script that uses :

/usr/bin/journalctl -u {unit name} -o short -f | sed 's/^/{unit name}/' | ncat {some-ip} {some port}

now i have the unit name in the begining of the log line ! and my log collector has unit names in the message !

example

journalctl -u mongodb.service -o short -f | sed 's/^/mongodb.service /' | ncat {some-ip} {some port}

will output :

mongodb Jun 26 09:11:35 localhost bash[1710]: 2014-06-26T09:11:35.714+0000 [rsHealthPoll] replset info mongodb-0:27017 heartbeat failed, retrying

Answer 4

Instead of _SYSTEMD_UNIT field use the container name.

I landed here because docker stack/swarm is missing the docker-compose logs feature to log all service on a single node.

With Docker Server Version: 18.03.0-ce and docker compose version 3.6 it is possible to send all logs to journald and log them by service name.In order to identify each container, tag them with the image name. See the logging section of docker-compose.dev.yml

Docker Compose:

version: '3.6'
networks:
    skynet:
    driver: overlay
services:
    mongo:
        image: mongo:3
        networks:
            - skynet
        volumes:
            - /data/mongodb:/data/db
        logging:
            driver: "journald"
            options:
                tag: "{{.Name}}"
                labels: "com.docker.stack.namespace"

journalctl:

sudo journalctl -b -o short --all -f COM_DOCKER_STACK_NAMESPACE=skynet 

Docker Command:

docker stack deploy -c docker-compose.dev.yml skynet

Will output: skynet_mongo.1.ins4s13luwiekrri3m5a8vwwl

Apr 24 14:23:08 c-wrk skynet_mongo.1.ins4s13luwiekrri3m5a8vwwl[19410]: 2018-04-24T12:23:08.212+0000 I NETWORK  [listener] connection accepted from 10.0.0.10:36518 #3 (3 connections now open)

Use full notes:

To view available labels use: docker inspect -f {{.Config.Labels}} <docker-id>

Available tags use: Customize log driver output

Config journald log driver: Docker Journald logging driver

Colorize log output: use journalctl | ccze -A