3
votes

I set up an EFK stack for gathering my different k8s pods logs based on this tutorial on a Microk8s single node cluster. Everything is up and working and I can connect kibanna to elasticsearch and see the indexes but in the discovery section of kibana there is no log related to my pods and there are kubelete logs.

When I checked the logs of fluentd I saw that it is full of backslashes:

2019-08-05 15:23:17 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: "2019-08-05T17:23:10.167379794+02:00 stdout P 2019-08-05 15:23:10 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \"2019-08-05T17:23:07.09726655+02:00 stdout P 2019-08-05 15:23:07 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \\\"2019-08-05T17:23:04.433817307+02:00 stdout P 2019-08-05 15:23:04 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \\\\\\\"2019-08-05T17:22:52.546188522+02:00 stdout P 2019-08-05 15:22:52 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \\\\\\\\\\\\\\\"2019-08-05T17:22:46.694679863+02:00 stdout F 
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

Can someone please tell me what I am doing wrong and where all those backslashes are coming from?

2

2 Answers

0
votes

There are few things you should check/try:

  1. Check how your patterns are configured (ex: td-agent.conf, fluent.conf).

  2. Test your format regex matching with Fluentular to see if there are any potential mistakes.

  3. Consider using fluent-plugin-multi-format-parser:

  4. For more information about resolving similar pattern not match issue, you can see this answer.

Please let me know if any of the above helped.

0
votes

For posterity, I'm going to provide this solution, which seems a lot more straight forward than anything else I've found:

env:
    - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
      value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/

Just place that in the env variables underneath your fluentd YAML configuration.