Logs not being flushed to Elasticsearch container through Fluentd

Question

I have a local setup running 2 conainers -

One for Elasticsearch (setup for development as detailed here - https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html). This I run as directed in the article using - docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.4.1

Another as a Fluentd aggregator (using this base image - https://hub.docker.com/r/fluent/fluentd/). My fluent.conf for testing purposes is as follows :

<source>
    @type forward
    port 24224
</source>

<match **>
    @type elasticsearch
    host 172.17.0.2    # Verified internal IP address of the ES container
    port 9200
    user elastic
    password changeme
    index_name fluentd
    buffer_type memory
    flush_interval 60
    retry_limit 17
    retry_wait 1.0
    include_tag_key true
    tag_key docker.test
    reconnect_on_error true
</match>

This I start with the command - docker run -p 24224:24224 -v /data:/fluentd/log vg/fluentd:latest

When I run my processes (that generate logs), and run these 2 containers, I see the following towards the end of stdout for the Fluentd container -

2017-06-15 12:16:33 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"172.17.0.2", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}

However, beyond this, I see no logs. When I login to http://localhost:9200 I only see the Elasticsearch welcome message.

I know the logs are reaching the Fluentd container, because when I change fluent.conf to redirect to a file, I see all the logs as expected. What am I doing wrong in my setup of Elasticsearch? How can I get to seeing all the indexes laid out correctly in my browser / through Kibana?

You always have the welcome message in that url. Have you tried to list the indexes? — Robert
This: curl 'localhost:9200/_cat/indices?v' elastic.co/guide/en/elasticsearch/reference/1.4/… — Robert
Blonde moment. Ok I see a bunch of indexes each with health yellow open and different uuids. How do I access each to see what it contains? — fwx
Ha ha. The yellow health is ok for dev (it means the lack of replicas). Make a post to /_search elastic.co/guide/en/elasticsearch/reference/current/search.html — Robert

Robert Robert · Accepted Answer · 2017-06-15T14:27:46

It seems that you are in the right track. Just check the indexes that were created in elasticsearch as follows:

curl 'localhost:9200/_cat/indices?v'

Docs: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/_list_all_indexes.html

There you can see each index name. So pick one and search within it:

curl 'localhost:9200/INDEXNAME/_search'

Docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html

However I recommend you to use kibana in order to have a better human experience. Just start it and by default it searches for an elastic in localhost. In the interface's config put the index name that you now know, and start to play with it.

Logs not being flushed to Elasticsearch container through Fluentd

1 Answers