I applied a fluentd-cloudwatch DeamonSet in EKS using config file as this: fluentd.yml
Situation ONE: with line 302 - line 313 commented in fluentd.yml, everything works well. I can see the new created log group and log streams in CLOUDWATCH PANEL.
Situation TWO: with line 302 - line 313 uncommented in fluentd.yml, fluentd cannot send any log data into CLOUDWATCH. log group and log streams are not created automatically.
I check the file /logs/fun-practice-log/server.log.0 exists in fluentd-cloudwatch pod.
The content in server.log.0 as below:
unknown - 2019-11-21 00:42:44,588 - WARNING - 404 GET /latest/dynamic/instance-identity/document (10.10.130.37) 0.38ms
unknown - 2019-11-21 00:42:44,818 - WARNING - 405 CONNECT 34.116.211.122:80 (10.10.130.217) 0.51ms
unknown - 2019-11-21 00:42:45,752 - WARNING - 405 CONNECT 34.116.211.122:80 (10.10.130.217) 0.64ms
f6c5a62f536a4444bbb12c0e45973141 - 2019-11-21 02:17:19,605 - INFO - 200 POST /fun_practice/v1.1/get_stat?user_token=seYBtp39Uko0lfsYoDBrQlZV8SWuF (10.0.15.188) 895.95ms
I want to stream the logs in folder /logs/fun-practice-log/ to CLOUDWATCH using source as below, I cannot figure out what I miss in config file.
<source>
@type tail
@id in_tail_svc
@label @hostlogs
path /logs/fun-practice-log/server.log.0
pos_file /logs/fun-practice-log/server.log.pos
tag host.svc
read_from_head true
refresh_interval 5
<parse>
@type regexp
expression /(?<request_id>[^ ]*) - (?<timestamp>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}) - (?<log_type>[^ ]*) - (?<message>.*)/
time_key timestamp
time_format %Y-%m-%d %H:%M:%S,%z
</parse>
</source>
