Fluentd Failing to connect to ElasticSearch cluster

Question

I have a local kubernetes cluster where I added a Fluentd Daemonset using the preconfigured elasticsearch image (fluent/fluentd-kubernetes-daemonset:elasticsearch). Step 2 of this article. I also have an elastic cluster running in the cloud. You can pass some env variables to the fluentd-elasticsearch image for configuration. It looks pretty straightforward, but when running the fluentd Pod I keep getting the error:

"Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"fa0acce34bf64db9bc9e46f98743c185.westeurope.azure.elastic-cloud.com\", :port=>9243, :scheme=>\"https\", :user=>\"username\", :password=>\"obfuscated\"})!" plugin_id="out_es"

when I try to reach the elastic cluster from within the pod with # wget https://fa0acce34bf64db9bc9e46f98743c185.westeurope.azure.elastic-cloud.com:9243/ I get a 401 unauthorized (cuz I havent submitted user/pass here), but it at least shows that the address is reachable.

Why is it failing to connect? I already set the FLUENT_ELASTICSEARCH_SSL_VERSION to 'TLSv1_2', i saw that that solved some problems for others.

Daemonset configuration:

kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-logging
  labels:
    app: fluentd
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
        env:
        - name:  FLUENT_ELASTICSEARCH_HOST
          value: "fa0acce34bf64db9bc9e46f98743c185.westeurope.azure.elastic-cloud.com"
        - name:  FLUENT_ELASTICSEARCH_PORT
          value: "9243"
        - name: FLUENT_ELASTICSEARCH_SCHEME
          value: "https"
        - name: FLUENT_UID
          value: "0"
        - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
          value: "false"
        - name: FLUENT_ELASTICSEARCH_SSL_VERSION
          value: "TLSv1_2"
        - name: FLUENT_ELASTICSEARCH_USER
          value: "<user>"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: "<password>"
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Are you sure that outbound port 9243 is opened? Have you tried to restart your DNS service? — Val
I can reach the es cluster from within my browser (where i get the user/pass promt) or from Postman. I can go to the address and it will show the cluster information, so it seems to be accessible from outside..? Also, it is a preconfigured trial from cloud.elastic.co where it should be as easy as to use the endpoint they provided. — Elfs
Yes, from your browser outside the pod, it's definitely ok, I tried it as well. But from inside the pod you can't apparently. hence why I'm asking whether you checked the iptables inside your pod and if the coredns service was running properly. — Val
I am a bit new to kubernetes, sorry if I misunderstood. I will read up on the ip tables and try your suggestion, thanks. — Elfs

Elfs Elfs · Accepted Answer · 2020-04-01T08:23:20

For anyone else who runs into this problem:

I was following a tutorial that used the 'image: fluent/fluentd-kubernetes-daemonset:elasticsearch' image. When you check their DockerHub (https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset) you can see that the :elaticsearch tag is a year old and probably outdated.

I changed the image for the DaemonSet to a more recent and stable tag 'fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch' and boom it works now.

Fluentd Failing to connect to ElasticSearch cluster

1 Answers