Fluentd installation instructions for AWS Elasticsearch Service

Question

We use AWS Elasticsearch Service. We would like to install fluentd to our Kubernetes cluster. Fluentd sends logs to AWS Elasticsearch Service. It there any instructions on how to do it? At this link, https://docs.fluentd.org/v/0.12/articles/kubernetes-fluentd, it shows steps on sending the logs to an Elasticsearch Pod. It means the elasticsearch is installed inside the Kubernetes cluster.

The link above shows make changes in fluentd-daemonset-elasticsearch.yaml (see below) for elasticsearch pod. What configuration changes I need to make for the AWS elasticsearch service?

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  ...
spec:
    ...
    spec:
      containers:
      - name: fluentd
        image: quay.io/fluent/fluentd-kubernetes-daemonset
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch-logging"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
            ......

You could try the solution from this GitHub link. And also some explanation from this article — Rohit

Rico Rico · Accepted Answer · 2020-08-19T01:59:18

Assuming that you have the required connectivity. For example, Kubernetes and Elasticsearch on the same VPC or if in different VPCs you have VPC peering configured.

Additionally, you have the firewall rules (Security Groups) to allow port 9200 from Kubernetes to the Elasticsearch cluster.

It should be straight forward (just point the configs to the AWS Elasticsearch cluster endpoint):

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  ...
spec:
    ...
    spec:
      containers:
      - name: fluentd
        image: quay.io/fluent/fluentd-kubernetes-daemonset
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "vpc-domain-name-identifier.region.es.amazonaws.com" 👈
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
            ......

This is if you are using VPCs. You can also use the public Elasticsearch endpoint too if you configured your cluster that way. (But it's less secure since you are going through the public cloud). Something like this:

https://search-domain-name-identifier.region.es.amazonaws.com

✌️

Fluentd installation instructions for AWS Elasticsearch Service

1 Answers