Apache response field name in filebeat

Question

I am using filebeat with ES as output. I have specified: input_type: log document_type: apache paths: - /var/log/httpd/*_log in /etc/filebeat/filebeat.yml and am able to successfully see results in Kibana. I am however playing around with "Watcher" and trying to create a watch based on an http return code of 404, I see no field in my Kibana filebeat results that corresponds to and only to "404", something like "response", I am afraid I am missing something because filebeat and ELK are BIG products, and help would be appreciated.

Answer 1

Filebeat forwards the log line "as is" in the message field of each event. In order to further process the message to extract details like response code into their own fields you can use Logstash.

In Logstash you would use the beats input to receive data from Filebeat, then apply a grok filter to parse the data from the message, and finally use an elasticsearch output to write the data to Elasticsearch.

Answer 2

An alternative to using Logstash is to use "Ingest Node", and a suitable pipeline in Elasticsearch.

https://www.elastic.co/guide/en/beats/filebeat/5.0/configuring-ingest-node.html

You can set up a pipeline that includes a Grok processor:

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/grok-processor.html

I did this with this pipeline.json file:

{
  "description": "Combined Apache Log Pipeline",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [ "%{COMBINEDAPACHELOG}" ]
      }
    }
  ]
}

I then ran this command to deploy the pipeline to the cluster:

curl -XPUT 'http://node-01.example.com:9200/_ingest/pipeline/combined-apache-log' [email protected]

Finally, I updated filebeat.yml to tell the Elasticsearch output to process events with the pipeline:

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts:
    - "192.168.0.1:9200"
    - "192.168.0.2:9200"
    - "192.168.0.3:9200"
  loadbalance: true
  pipeline: combined-apache-log

This appears to be working without needing Logstash.

I'm definitely seeing fields for response, referrer, response, etc.