I am new to ELK Stack and unable to get basic working. Please help.
Steps Done:
- Started elasticsearch.bat (Started Fine)
Open http://localhost:9200 in Chrome and got:
{ "name" : "SrfDQVN", "cluster_name" : "elasticsearch", "cluster_uuid" : "ZtUOrpL8S8q5gAdh6tCS4g", "version" : { "number" : "6.3.1", "build_flavor" : "default", "build_type" : "zip", "build_hash" : "eb782d0", "build_date" : "2018-06-29T21:59:26.107521Z", "build_snapshot" : false, "lucene_version" : "7.3.1", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }Logstash sample.conf file in logstash dir
-sample.conf
input {
beats {
type => beats
port => 5044
host => "localhost"
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_IS08601:timestamp_string}%{SPACE}%{GREEDYDATA:line}"]
}
}
date {
match => [ "timestamp_string" , "IS08601" ]
}
mutate {
remove_field => [message, timestamp_string]
}
}
output {
stdout {
codec => rubydebug
}
}
logstash.yml
http.host: "127.0.0.1" http.port: 5044Started logstash using command from logstash directory
bin\logstash -f sample.confHere is the last line from the command window after starting logstash
[2018-08-01T23:54:00,246][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>5044}Filebeat sample.log file
2018-08-01T13:00:00Z I met a traveller from an antique land 1111 2018-08-01T14:00:00Z I met a traveller from an antique land 2222 2018-08-01T15:00:00Z I met a traveller from an antique land 3333 2018-08-01T16:00:00Z I met a traveller from an antique land 4444 2018-08-01T17:00:00Z I met a traveller from an antique land 5555 2018-08-01T18:00:00Z I met a traveller from an antique land 6666 2018-08-01T19:00:00Z I met a traveller from an antique land 7777 2018-08-01T20:00:00Z I met a traveller from an antique land 8888 2018-08-01T21:00:00Z I met a traveller from an antique land 9999 2018-08-01T22:00:00Z I met a traveller from an antique land 11111 2018-08-01T23:00:00Z I met a traveller from an antique land 22222 2018-08-01T00:00:00Z I met a traveller from an antique land 33333 2018-09-01T01:00:00Z I met a traveller from an antique land 44444 2018-09-01T02:00:00Z I met a traveller from an antique land 55555 2018-09-01T03:00:00Z I met a traveller from an antique land 66666 2018-09-01T04:00:00Z I met a traveller from an antique land 77777filebeat.yml file
filebeat.prospectors: - type: log enabled: true paths: - .\sample.log output.logstash: hosts: ["localhost:5044"]Running following command to start filebeat through elevated command prompt
PS C:\Program Files\filebeat-6.3.2-windows-x86_64> .\filebeat.exe -c .\filebeat.yml -e -v
I get the following error:
2018-08-02T00:25:22.445+0530 INFO instance/beat.go:492 Home path: [C:\Program Files\filebeat-6.3.2-windows-x86_64] Config path: [C:\Program Files\filebeat-6.3.2-windows-x86_64] Data path: [C:\Program Files\filebeat-6.3.2-windows-x86_64\data] Logs path: [C:\Program Files\filebeat-6.3.2-windows-x86_64\logs]
2018-08-02T00:25:22.446+0530 INFO instance/beat.go:499 Beat UUID: eddf3426-d568-4252-8165-ff358472d0a4
2018-08-02T00:25:22.447+0530 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "C:\\Program Files\\filebeat-6.3.2-windows-x86_64", "data": "C:\\Program Files\\filebeat-6.3.2-windows-x86_64\\data", "home": "C:\\Program Files\\filebeat-6.3.2-windows-x86_64", "logs": "C:\\Program Files\\filebeat-6.3.2-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "eddf3426-d568-4252-8165-ff358472d0a4"}}}
2018-08-02T00:25:22.447+0530 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "45a9a9e1561b6c540e94211ebe03d18abcacae55", "libbeat": "6.3.2", "time": "2018-07-20T04:17:39.000Z", "version": "6.3.2"}}}
2018-08-02T00:25:22.448+0530 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
2018-08-02T00:25:22.470+0530 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
2018-08-02T00:25:22.470+0530 INFO pipeline/module.go:81 Beat name: LP1-AP-51683797
2018-08-02T00:25:22.471+0530 WARN [cfgwarn] beater/filebeat.go:61 DEPRECATED: prospectors are deprecated, Use `inputs` instead. Will be removed in version: 7.0.0
2018-08-02T00:25:22.480+0530 INFO instance/beat.go:315 filebeat start running.
2018-08-02T00:25:22.480+0530 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-08-02T00:25:22.481+0530 INFO registrar/registrar.go:117 Loading registrar data from C:\Program Files\filebeat-6.3.2-windows-x86_64\data\registry
2018-08-02T00:25:22.490+0530 INFO registrar/registrar.go:124 States Loaded from registrar: 2
2018-08-02T00:25:22.492+0530 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-02T00:25:22.493+0530 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-08-02T00:25:22.502+0530 INFO log/input.go:118 Configured paths: [C:\Program Files\filebeat-6.3.2-windows-x86_64\sample.log]
2018-08-02T00:25:22.503+0530 INFO input/input.go:88 Starting input of type: log; ID: 6142389514979083186
2018-08-02T00:25:22.510+0530 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-08-02T00:25:22.511+0530 INFO log/harvester.go:228 Harvester started for file: C:\Program Files\filebeat-6.3.2-windows-x86_64\sample.log
2018-08-02T00:25:23.532+0530 ERROR logstash/async.go:235 Failed to publish events caused by: lumberjack protocol error
2018-08-02T00:25:23.535+0530 ERROR logstash/async.go:235 Failed to publish events caused by: client is not connected
2018-08-02T00:25:24.535+0530 ERROR pipeline/output.go:92 Failed to publish events: client is not connected
2018-08-02T00:25:25.370+0530 INFO beater/filebeat.go:420 Stopping filebeat
2018-08-02T00:25:25.370+0530 INFO crawler/crawler.go:109 Stopping Crawler
2018-08-02T00:25:25.371+0530 INFO crawler/crawler.go:119 Stopping 1 inputs
2018-08-02T00:25:25.371+0530 INFO input/input.go:122 input ticker stopped
2018-08-02T00:25:25.378+0530 INFO input/input.go:139 Stopping Input: 6142389514979083186
2018-08-02T00:25:25.379+0530 INFO log/harvester.go:249 Reader was closed: C:\Program Files\filebeat-6.3.2-windows-x86_64\sample.log. Closing.
2018-08-02T00:25:25.379+0530 INFO crawler/crawler.go:135 Crawler stopped
2018-08-02T00:25:25.380+0530 INFO registrar/registrar.go:339 Stopping Registrar
2018-08-02T00:25:25.382+0530 INFO registrar/registrar.go:265 Ending Registrar
2018-08-02T00:25:25.541+0530 ERROR logstash/async.go:235 Failed to publish events caused by: lumberjack protocol error
2018-08-02T00:25:25.542+0530 ERROR logstash/async.go:235 Failed to publish events caused by: client is not connected
2018-08-02T00:25:25.631+0530 INFO [monitoring] log/log.go:132 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":156,"time":{"ms":156},"value":156},"user":{"ticks":31,"time":{"ms":31}}},"info":{"ephemeral_id":"b45276ef-6cc2-40b7-8c4d-4e099beedab2","uptime":{"ms":3010}},"memstats":{"gc_next":5746432,"memory_alloc":4250656,"memory_total":6135552,"rss":23027712}},"filebeat":{"events":{"active":15,"added":17,"done":2},"harvester":{"closed":1,"open_files":0,"running":0,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":30,"total":30},"read":{"bytes":6},"type":"logstash","write":{"bytes":778}},"pipeline":{"clients":0,"events":{"active":15,"filtered":2,"published":15,"retry":45,"total":17}}},"registrar":{"states":{"current":2,"update":2},"writes":{"success":3,"total":3}},"system":{"cpu":{"cores":8}}}}}
2018-08-02T00:25:25.631+0530 INFO [monitoring] log/log.go:133 Uptime: 3.2490036s
2018-08-02T00:25:25.633+0530 INFO [monitoring] log/log.go:110 Stopping metrics logging.
2018-08-02T00:25:25.634+0530 INFO instance/beat.go:321 filebeat stopped.
