fluentd not showing output of python script

Question

I'm trying to run python script in fluentd. Python script simply writes some json to stdout.

Python script (events.py) :-

import sys
import json

for i in range(3):
    sys.stdout.write(json.dumps({"a":i,"b":i}))
    sys.stdout.write("\n")

fluentd suppose to execute the script and write output of the python program to stdout as I'm using exec input plugin and stdout output plugin.

fluentd config:- (events.py is in current directory)

<source>
  @type exec
  tag pythonevents
  command python events.py
  run_interval 1m
  <parse>
    @type json
  </parse>
</source>

<match pythonevents> 
  @type stdout
</match>

and the output when i run fluentd:-

fluentd -c /fluentd/etc/main-config.conf
2021-06-19 02:36:08 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/main-config.conf"
2021-06-19 02:36:08 +0000 [info]: gem 'fluent-plugin-pgjson' version '1.0.1'  
2021-06-19 02:36:08 +0000 [info]: gem 'fluent-plugin-postgres' version '0.1.0'
2021-06-19 02:36:08 +0000 [info]: gem 'fluent-plugin-sql' version '2.2.0'     
2021-06-19 02:36:08 +0000 [info]: gem 'fluentd' version '1.13.0'
2021-06-19 02:36:08 +0000 [info]: gem 'fluentd' version '1.12.4'
2021-06-19 02:36:08 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type exec
    tag "pythonevents"
    command "python events.py"
    run_interval 1m
    <parse>
      @type "json"
    </parse>
  </source>
  <match pythonevents>
    @type stdout
  </match>
</ROOT>
2021-06-19 02:36:08 +0000 [info]: starting fluentd-1.13.0 pid=7 ruby="2.6.7"
2021-06-19 02:36:08 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/main-config.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2021-06-19 02:36:09 +0000 [info]: adding match pattern="pythonevents" type="stdout"
2021-06-19 02:36:09 +0000 [info]: adding source type="exec"
2021-06-19 02:36:09 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
2021-06-19 02:36:09 +0000 [info]: #0 fluentd worker is now running worker=0

Only this much output comes. So the issue is that fluentd is not showing expected output in stdout.

EDIT :: I'm using docker.

Sam A. Sam A. · Accepted Answer · 2021-07-22T11:30:20

This configuration does work for me.

Maybe you should use absolute path for your python binary and script.

command /my/path/python /my/script/path/events.py

The first execution of the command is after the first run_interval.

fluentd not showing output of python script

1 Answers