Kibana not connecting to Elasticsearch (["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"})

Question

Context: I have been struggling this whole week to get this stack up and running: filebeat -> kafka -> logstash -> elasticsearch - kibana, each one in its own docker (you will find around 3 or 4 other questions mine here without answer resulted from different tentatives). I have decided to downsize the stack and then move block by block untill i reach a final docker-compose. Then I tried the simplest stack I can imagine to push forward the simplest log I can imagine and I am facing the issue mentioned above in my question.

Issue: I am trying to run straight from command line three docker containers: filebeat, elasticsearch and kibana. When I try to start kibana I get "No living connections". I am following carefully the answer provide in another stackoverflow question. Any clue why I am not able to connect from Kibana container to Elasticsearch container?

Here are all three docker commands:

docker run -d -p 9200:9200 -e "discovery.type=single-node" --volume C:\Dockers\simplest-try\esdata:/usr/share/elasticsearch/data --name elasticsearch_container docker.elastic.co/elasticsearch/elasticsearch:7.5.2

docker run -d --mount type=bind,source=C:\Dockers\simplest-try\filebeat.yml,target=/usr/share/filebeat/filebeat.yml --volume C:\Dockers\simplest-try\mylogs:/mylogs docker.elastic.co/beats/filebeat:7.5.2 

docker run -d --name kibana -p 5601:5601 --link elasticsearch_container:elasticsearch_alias -e "ELASTICSEARCH_URL=http://elasticsearch_alias:9200" docker.elastic.co/kibana/kibana:7.5.2

ElasticSearch is up and running:

C:\Dockers\simplest-try>curl localhost:9200
{
  "name" : "ffaa2d39a8b2",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "QWYLaAqwSqu76fNwFtZ5AA",
  "version" : {
    "number" : "7.5.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
    "build_date" : "2020-01-15T12:11:52.313576Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Kibana container console:

{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [security,licensing,code,timelion,features,spaces,translations,uiActions,newsfeed,inspector,embeddable,advancedUiActions,expressions,eui_utils,data]"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:25Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-02-06T14:53:41Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-02-06T14:53:42Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2020-02-06T14:53:43Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/.kibana => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2020-02-06T14:53:43Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/.kibana_task_manager => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2020-02-06T14:53:44Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-02-06T14:53:44Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-02-06T14:53:44Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: No Living connections"}

Although not straigt to my question title, here are details about Filebeat: Filebeat try to harverst my log files

2020-02-06T14:32:23.782Z    INFO    crawler/crawler.go:72   Loading Inputs: 1
2020-02-06T14:32:23.782Z    INFO    log/input.go:152    Configured paths: [/mylogs/*.log]
2020-02-06T14:32:23.782Z    INFO    input/input.go:114  Starting input of type: log; ID: 4094557846902174710 
2020-02-06T14:32:23.782Z    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2020-02-06T14:32:23.788Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/y.log
2020-02-06T14:32:23.790Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/x.log

filebeat.yml

filebeat.inputs:
- type: log
  paths:
  - '/mylogs/*.log'
  json.message_key: log
  json.keys_under_root: true
  processors:
  - add_docker_metadata: ~
output.elasticsearch:
  hosts: ["localhost:9200"]

*** edited

logs after Ibexit's suggestion

2020-02-12T21:33:03.575Z    INFO    instance/beat.go:610    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-12T21:33:03.588Z    INFO    instance/beat.go:618    Beat ID: d0c71c07-23e0-44e5-b497-195ee9552fe8
2020-02-12T21:33:03.588Z    INFO    [seccomp]   seccomp/seccomp.go:124  Syscall filter successfully installed
2020-02-12T21:33:03.588Z    INFO    [beat]  instance/beat.go:941    Beat info   {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "d0c71c07-23e0-44e5-b497-195ee9552fe8"}}}
2020-02-12T21:33:03.588Z    INFO    [beat]  instance/beat.go:950    Build info  {"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-02-12T21:33:03.588Z    INFO    [beat]  instance/beat.go:953    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.12"}}}
2020-02-12T21:33:03.590Z    INFO    [beat]  instance/beat.go:957    Host info   {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-12T20:32:39Z","containerized":true,"name":"fcfaea4080e7","ip":["127.0.0.1/8","172.17.0.3/16"],"kernel_version":"4.19.76-linuxkit","mac":["02:42:ac:11:00:03"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-12T21:33:03.590Z    INFO    [beat]  instance/beat.go:986    Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-12T21:33:02.690Z"}}}
2020-02-12T21:33:03.590Z    INFO    instance/beat.go:297    Setup Beat: filebeat; Version: 7.5.2
2020-02-12T21:33:03.590Z    INFO    [index-management]  idxmgmt/std.go:182  Set output.elasticsearch.index to 'filebeat-7.5.2' as ILM is enabled.
2020-02-12T21:33:03.591Z    INFO    elasticsearch/client.go:171 Elasticsearch url: http://elasticsearch:9200
2020-02-12T21:33:03.591Z    INFO    [publisher] pipeline/module.go:97   Beat name: fcfaea4080e7
2020-02-12T21:33:03.593Z    INFO    instance/beat.go:429    filebeat start running.
2020-02-12T21:33:03.593Z    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-02-12T21:33:03.594Z    INFO    registrar/migrate.go:104    No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-02-12T21:33:03.594Z    INFO    registrar/migrate.go:112    Initialize registry meta file
2020-02-12T21:33:03.600Z    INFO    registrar/registrar.go:108  No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-02-12T21:33:03.611Z    INFO    registrar/registrar.go:145  Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-12T21:33:03.611Z    INFO    registrar/registrar.go:152  States Loaded from registrar: 0
2020-02-12T21:33:03.612Z    INFO    crawler/crawler.go:72   Loading Inputs: 1
2020-02-12T21:33:03.612Z    INFO    log/input.go:152    Configured paths: [/mylogs/*.log]
2020-02-12T21:33:03.612Z    INFO    input/input.go:114  Starting input of type: log; ID: 4094557846902174710 
2020-02-12T21:33:03.612Z    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2020-02-12T21:33:03.640Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/b.log
2020-02-12T21:33:03.640Z    ERROR   readjson/json.go:52 Error decoding JSON: invalid character '\'' looking for beginning of object key string
2020-02-12T21:33:03.642Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/c.log
2020-02-12T21:33:03.644Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/w.log
2020-02-12T21:33:03.645Z    ERROR   readjson/json.go:52 Error decoding JSON: invalid character 'q' looking for beginning of value
2020-02-12T21:33:03.645Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/x.log
2020-02-12T21:33:03.652Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/y.log
2020-02-12T21:33:04.654Z    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticsearch:9200))
2020-02-12T21:33:04.684Z    INFO    elasticsearch/client.go:753 Attempting to connect to Elasticsearch version 7.5.2
2020-02-12T21:33:04.720Z    INFO    [index-management]  idxmgmt/std.go:256  Auto ILM enable success.
2020-02-12T21:33:04.724Z    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false
2020-02-12T21:33:04.724Z    INFO    [index-management]  idxmgmt/std.go:269  ILM policy successfully loaded.
2020-02-12T21:33:04.725Z    INFO    [index-management]  idxmgmt/std.go:408  Set setup.template.name to '{filebeat-7.5.2 {now/d}-000001}' as ILM is enabled.
2020-02-12T21:33:04.725Z    INFO    [index-management]  idxmgmt/std.go:413  Set setup.template.pattern to 'filebeat-7.5.2-*' as ILM is enabled.
2020-02-12T21:33:04.725Z    INFO    [index-management]  idxmgmt/std.go:447  Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.5.2 {now/d}-000001} as ILM is enabled.
2020-02-12T21:33:04.725Z    INFO    [index-management]  idxmgmt/std.go:451  Set settings.index.lifecycle.name in template to {filebeat-7.5.2 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-02-12T21:33:04.730Z    INFO    template/load.go:89 Template filebeat-7.5.2 already exists and will not be overwritten.
2020-02-12T21:33:04.730Z    INFO    [index-management]  idxmgmt/std.go:293  Loaded index template.
2020-02-12T21:33:04.734Z    INFO    [index-management]  idxmgmt/std.go:304  Write alias successfully generated.
2020-02-12T21:33:04.736Z    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
2020-02-12T21:33:33.595Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":50}},"total":{"ticks":100,"time":{"ms":107},"value":100},"user":{"ticks":50,"time":{"ms":57}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":30060}},"memstats":{"gc_next":8351264,"memory_alloc":4760176,"memory_total":12037984,"rss":43970560},"runtime":{"goroutines":42}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":5,"running":5,"started":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":3,"batches":1,"total":3},"read":{"bytes":2942},"type":"elasticsearch","write":{"bytes":2545}},"pipeline":{"clients":1,"events":{"active":0,"filtered":5,"published":3,"retry":3,"total":8},"queue":{"acked":3}}},"registrar":{"states":{"current":5,"update":8},"writes":{"success":7,"total":7}},"system":{"cpu":{"cores":2},"load":{"1":0.02,"15":0.08,"5":0.1,"norm":{"1":0.01,"15":0.04,"5":0.05}}}}}}
2020-02-12T21:33:58.657Z    ERROR   readjson/json.go:52 Error decoding JSON: invalid character 'E' looking for beginning of value
2020-02-12T21:33:58.657Z    ERROR   readjson/json.go:52 Error decoding JSON: invalid character 'a' looking for beginning of value
2020-02-12T21:34:03.596Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":13}},"total":{"ticks":120,"time":{"ms":16},"value":120},"user":{"ticks":60,"time":{"ms":3}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":60059}},"memstats":{"gc_next":8351264,"memory_alloc":5345000,"memory_total":12622808},"runtime":{"goroutines":42}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2,"batches":1,"total":2},"read":{"bytes":351},"write":{"bytes":1062}},"pipeline":{"clients":1,"events":{"active":0,"published":2,"total":2},"queue":{"acked":2}}},"registrar":{"states":{"current":5,"update":2},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.01,"15":0.08,"5":0.09,"norm":{"1":0.005,"15":0.04,"5":0.045}}}}}}
2020-02-12T21:34:33.599Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":10}},"total":{"ticks":130,"time":{"ms":14},"value":130},"user":{"ticks":60,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":90059}},"memstats":{"gc_next":8351264,"memory_alloc":5714936,"memory_total":12992744,"rss":380928},"runtime":{"goroutines":42}},"filebeat":{"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":5}},"system":{"load":{"1":0.07,"15":0.08,"5":0.1,"norm":{"1":0.035,"15":0.04,"5":0.05}}}}}}
2020-02-12T21:34:33.686Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/d.log
2020-02-12T21:35:03.597Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":16}},"total":{"ticks":140,"time":{"ms":21},"value":140},"user":{"ticks":60,"time":{"ms":5}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":120059}},"memstats":{"gc_next":8351264,"memory_alloc":6130552,"memory_total":13408360},"runtime":{"goroutines":46}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":6,"running":6,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":6,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.15,"15":0.09,"5":0.12,"norm":{"1":0.075,"15":0.045,"5":0.06}}}}}}
2020-02-12T21:35:33.596Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":14}},"total":{"ticks":170,"time":{"ms":23},"value":170},"user":{"ticks":70,"time":{"ms":9}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":150060}},"memstats":{"gc_next":7948720,"memory_alloc":4110408,"memory_total":13866968},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.09,"15":0.08,"5":0.11,"norm":{"1":0.045,"15":0.04,"5":0.055}}}}}}
2020-02-12T21:36:03.597Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":7}},"total":{"ticks":190,"time":{"ms":9},"value":190},"user":{"ticks":80,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":180059}},"memstats":{"gc_next":7948720,"memory_alloc":4399584,"memory_total":14156144},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.38,"15":0.11,"5":0.18,"norm":{"1":0.19,"15":0.055,"5":0.09}}}}}}
2020-02-12T21:36:33.596Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":120,"time":{"ms":11}},"total":{"ticks":200,"time":{"ms":15},"value":200},"user":{"ticks":80,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":210059}},"memstats":{"gc_next":7948720,"memory_alloc":4776320,"memory_total":14532880},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.23,"15":0.1,"5":0.16,"norm":{"1":0.115,"15":0.05,"5":0.08}}}}}}
2020-02-12T21:37:03.600Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":120,"time":{"ms":9}},"total":{"ticks":210,"time":{"ms":16},"value":210},"user":{"ticks":90,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":240059}},"memstats":{"gc_next":7948720,"memory_alloc":5142416,"memory_total":14898976},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.14,"15":0.1,"5":0.14,"norm":{"1":0.07,"15":0.05,"5":0.07}}}}}}
2020-02-12T21:37:33.596Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":12}},"total":{"ticks":240,"time":{"ms":24},"value":240},"user":{"ticks":100,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":270060}},"memstats":{"gc_next":7946160,"memory_alloc":4111832,"memory_total":15348288},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.08,"15":0.09,"5":0.13,"norm":{"1":0.04,"15":0.045,"5":0.065}}}}}}
2020-02-12T21:38:03.596Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":11}},"total":{"ticks":250,"time":{"ms":12},"value":250},"user":{"ticks":100,"time":{"ms":1}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":300060}},"memstats":{"gc_next":7946160,"memory_alloc":4489960,"memory_total":15726416},"runtime":{"goroutines":46}},"filebeat":{"harvester":{"open_files":6,"running":6}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.1,"15":0.09,"5":0.13,"norm":{"1":0.05,"15":0.045,"5":0.065}}}}}}
2020-02-12T21:38:08.676Z    INFO    log/harvester.go:276    File is inactive: /mylogs/w.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:08.676Z    INFO    log/harvester.go:276    File is inactive: /mylogs/c.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:08.678Z    INFO    log/harvester.go:276    File is inactive: /mylogs/b.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:08.678Z    INFO    log/harvester.go:276    File is inactive: /mylogs/y.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:38:13.706Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/y.log
2020-02-12T21:38:33.594Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":5}},"total":{"ticks":250,"time":{"ms":9},"value":250},"user":{"ticks":100,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":330059}},"memstats":{"gc_next":7946160,"memory_alloc":5014240,"memory_total":16250696},"runtime":{"goroutines":34}},"filebeat":{"events":{"added":5,"done":5},"harvester":{"closed":4,"open_files":3,"running":3,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":5,"total":5}}},"registrar":{"states":{"current":6,"update":5},"writes":{"success":5,"total":5}},"system":{"load":{"1":0.88,"15":0.15,"5":0.31,"norm":{"1":0.44,"15":0.075,"5":0.155}}}}}}
2020-02-12T21:39:03.595Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":6}},"total":{"ticks":270,"time":{"ms":8},"value":270},"user":{"ticks":110,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":360059}},"memstats":{"gc_next":7946160,"memory_alloc":5284712,"memory_total":16521168},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":6}},"system":{"load":{"1":0.68,"15":0.16,"5":0.31,"norm":{"1":0.34,"15":0.08,"5":0.155}}}}}}
2020-02-12T21:39:03.676Z    INFO    log/harvester.go:276    File is inactive: /mylogs/x.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:39:33.596Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":5}},"total":{"ticks":270,"time":{"ms":12},"value":270},"user":{"ticks":110,"time":{"ms":7}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":390059}},"memstats":{"gc_next":7666032,"memory_alloc":3879448,"memory_total":16793464},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"closed":1,"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":6,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.48,"15":0.16,"5":0.3,"norm":{"1":0.24,"15":0.08,"5":0.15}}}}}}
2020-02-12T21:39:38.705Z    INFO    log/harvester.go:276    File is inactive: /mylogs/d.log. Closing because close_inactive of 5m0s reached.
2020-02-12T21:39:43.714Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/d.log
2020-02-12T21:39:43.715Z    ERROR   readjson/json.go:52 Error decoding JSON: EOF
2020-02-12T21:39:49.724Z    INFO    log/harvester.go:264    File was truncated. Begin reading file from offset 0: /mylogs/d.log
2020-02-12T21:39:53.720Z    INFO    log/harvester.go:251    Harvester started for file: /mylogs/d.log
2020-02-12T21:39:53.721Z    ERROR   readjson/json.go:52 Error decoding JSON: EOF
2020-02-12T21:40:03.597Z    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":190,"time":{"ms":30}},"total":{"ticks":320,"time":{"ms":46},"value":320},"user":{"ticks":130,"time":{"ms":16}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"d28c1982-c6bd-43b4-bfbb-c439f909b057","uptime":{"ms":420059}},"memstats":{"gc_next":7666032,"memory_alloc":4930512,"memory_total":17844528},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"closed":2,"open_files":2,"running":2,"started":2},"input":{"log":{"files":{"truncated":1}}}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":4,"batches":2,"total":4},"read":{"bytes":702},"write":{"bytes":2270}},"pipeline":{"clients":1,"events":{"active":0,"filtered":4,"published":4,"total":8},"queue":{"acked":4}}},"registrar":{"states":{"current":6,"update":8},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.59,"15":0.17,"5":0.33,"norm":{"1":0.295,"15":0.085,"5":0.165}}}}}}

Your kibana somehow try to connect http://elasticsearch:9200/ instead of http://elasticsearch_alias:9200 — alpert

ibexit ibexit · Accepted Answer · 2020-02-07T22:36:12

The Problem is that the three container are separated in terms of networking from each other and/or misconfigured. Let us discuss what is actualy happening and how to fix it:

1. Elasticsearch

You are starting an elasticsearch container named elasticsearch_container:

docker run -d -p 9200:9200 -e "discovery.type=single-node" --volume C:\Dockers\simplest-try\esdata:/usr/share/elasticsearch/data --name elasticsearch_container docker.elastic.co/elasticsearch/elasticsearch:7.5.2

So far, so good.

2. Filebeat

As mentioned at the beginning, the containers are separated from each other. In order to make elasticsearch visible for filebeat, you need to create a link:

docker run -d --link elasticsearch_container:elasticsearch --mount type=bind,source=C:\Dockers\simplest-try\filebeat.yml,target=/usr/share/filebeat/filebeat.yml --volume C:\Dockers\simplest-try\mylogs:/mylogs docker.elastic.co/beats/filebeat:7.5.2

Please note the container link: --link elasticsearch_container:elasticsearch which is the key here. Now, as the elasticsearch_container is visible to filebeat under the name elasticsearch, we need to change the filebeat.yml in that way:

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

Using localhost here is meant from the perspective of the filebeat container, which is unaware of the docker host. So localhost within the filebeat container adresses the filebeat container itself. But with the configuration change above, we changed it to the name of the linked elasticsearch container, what should do the trick.

3. Kibana

Kibana is complaining about missing connection to elasticsearch:

Unable to revive connection: http://elasticsearch:9200

Here it's the same case as for filebeat: elasticsearch is not visible to the kibana container under the name elasticsearch but elasticsearch_alias. Additionally, ELASTICSEARCH_URL is not an expected configuration in the version you are using. elasticsearch.hosts is the correct setting and defaults to http://elasticsearch:9200. And this is the root of the error message: kibana is not recognising ELASTICSEARCH_URL, falls back to the default value and fails because elasticsearch_container is linked as elasticsearch_alias and not as elasticsearch. Fixing this is easy, as we need just to remove ELASTICSEARCH_URL and let kibana fall back to the default. To make elasticsearch visible to kibana, we just apply the same link as we did for filebeat already:

docker run -d --name kibana -p 5601:5601 --link elasticsearch_container:elasticsearch docker.elastic.co/kibana/kibana:7.5.2

Important: Please dispose (stop & remove) the old container instances as they are claiming the container names before executing the discussed changes.

Kibana not connecting to Elasticsearch (["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"})

1 Answers