EKS pods logging to Elastic Cloud

Question

I am trying to set up pods logs shipping from EKS to ElasticSearch Cloud.

According to Fluent Bit for Amazon EKS on AWS Fargate is here, ElasticSearch should be supported:

You can choose between CloudWatch, Elasticsearch, Kinesis Firehose and Kinesis Streams as outputs.

According to FluentBit Configuration Parameters for ElasticSearch having Cloud_ID and Cloud_Auth parameters should be enough to ship logs to Elasticsearch Cloud.

An example here shows how to configure ES output for FluentBit, so my config looks like:

[OUTPUT]
  Name  es
  Match *
  Logstash_Format On
  Logstash_Prefix ${logstash_prefix}
  tls On
  tls.verify Off
  Pipeline date_to_timestamp
  Cloud_ID ${es_cloud_id}
  Cloud_Auth ${es_cloud_auth}
  Trace_Output On

I am running a simple ngnix container to generate some logs (as in one of the linked examples), but they don't seem to appear in my ElasticSearch / Kibana.

Am I missing anything? How do I ship logs to ElasticSearch Cloud?

Also, Trace_Output On is supposed to log FluentBits' attempts to ship logs, but where can I see these logs on EKS?

I assume you are running nginx in a Fargate Profile right? In which Namespace have you created the FluentBit ConfigMap? What resource do you use to start nginx? Can you confirm that the nginx pod has a side car container automatically created? — Bastian Klein

lennart lennart · Accepted Answer · 2021-05-28T08:08:14

I also ran into this. It seems to me only AWS ElasticSearch is supported when using the AWS managed FluentBit (from what I can tell).

https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-eks-adds-built-in-logging-support-for-aws-fargate/

You can work around this by using a sidecar fluentbit container (which can send to ElasticSearch) if that's an option for you. You will need to modify the application to have logs written to the filesystem.

Or you can use the managed FluentBit with the cloudwatch output, subscribe with to the log group with a lambda function and send it to ES.

EKS pods logging to Elastic Cloud

1 Answers