geoip lookup failure elastic stack logstash

Question

Using filebeat to send apache logs from Windows System and to my logstash server in linux EC2 and then to elastic search and Kibana.

Elastic search and Kibana - 5.3 Logstash and filebeat - 5.3

filebeat.yml :

filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - C:\Users\Sagar\Desktop\elastic_test4\data\log\*

output.logstash:
  # The Logstash hosts
  hosts: ["10.101.00.11:5044"]
  template.name: "filebeat-poc"
  template.path: "filebeat.template.json"
  template.overwrite: false

logstash.conf in Ubuntu Linux EC2 instance

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
      match => {
        "message" => "%{COMBINEDAPACHELOG}"
      }
  }
  geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
  }
   mutate {
      convert => [ "[geoip][coordinates]", "float"]
  }
 }
output {
  elasticsearch {
  hosts => ["elastic-instance-1.es.amazonaws.com:80"]
  index => "apache-%{+YYYY.MM.dd}"
  document_type => "apache_logs"
 }
  stdout { codec => rubydebug }
}

my dummy log file.

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846
64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] "GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 4523
64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Mar/2004:16:11:58 -0800] "GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200 7352
64.242.88.10 - - [07/Mar/2004:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
64.242.88.10 - - [07/Mar/2004:16:23:12 -0800] "GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.12¶m2=1.12 HTTP/1.1" 200 11382
64.242.88.10 - - [07/Mar/2004:16:24:16 -0800] "GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1" 200 4924
64.242.88.10 - - [07/Mar/2004:16:29:16 -0800] "GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12851
64.242.88.10 - - [07/Mar/2004:16:30:29 -0800] "GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1" 401 12851
64.242.88.10 - - [07/Mar/2004:16:31:48 -0800] "GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1" 200 3732
64.242.88.10 - - [07/Mar/2004:16:32:50 -0800] "GET /twiki/bin/view/Main/WebChanges HTTP/1.1" 200 40520
64.242.88.10 - - [07/Mar/2004:16:33:53 -0800] "GET /twiki/bin/edit/Main/Smtpd_etrn_restrictions?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12851

I am able to send those logs to elastic and kibana dashboard. Pipeline is setup and its working but geoip is not working.

This is my kibana output on search.

{
        "_index": "apache-2017.06.15",
        "_type": "apache_logs",
        "_id": "AVyqJhi6ItD-cRj2_AW6",
        "_score": 1,
        "_source": {
          "@timestamp": "2017-06-15T05:06:48.038Z",
          "offset": 154,
          "@version": "1",
          "input_type": "log",
          "beat": {
            "hostname": "sagar-machine",
            "name": "sagar-machine",
            "version": "5.3.2"
          },
          "host": "by-df164",
          "source": """C:\Users\Sagar\Desktop\elastic_test4\data\log\apache-log.log""",
          "message": """64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846""",
          "type": "log",
          "tags": [
            "beats_input_codec_plain_applied",
            "_grokparsefailure",
            "_geoip_lookup_failure"
          ]
        }
      }

Any idea why I am facing this issue.

Answer 1

You have a _grokparsefailure, therefore the clientip field doesn't exist. This causes the _geoip_lookup_failure, because the geoip filter is sourcing the clientip field which doesn't exist.

Your logs match the %{COMMONAPACHELOG} pattern instead of the one you are using. So your config would look like:

filter {
  grok {
      match => {
        "message" => "%{COMMONAPACHELOG}"
      }
   }
   ...
}

After using the correct pattern, you should notice the clientip field exists and after that, hopefully the geoip filter will work. :)

Answer 2

i don't know your logs format is correct or not for apache. because your logs are looking like this

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

and standard apache logs look like this

149.148.126.144 - - [10/Sep/2017:06:30:44 -0700] "GET /apps/cart.jsp?appID=6944 HTTP/1.0" 200 4981 "http://hernandez.net/app/main/search/homepage.php" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5322 (KHTML, like Gecko) Chrome/13.0.896.0 Safari/5322"

I'm recommending please standardized your coming apache logs format. otherwise default grok configuration will not work for you. and then you have to write your own grok pattern for your custom logs. which will parse your coming log lines

apart from that there are many reasons you are getting such errors

you didn't commented 'filebeat-template' in your filebeat configuration. filebeat template we will use when you directly sending logs from filebeat to elastic.

change configuration for your filebeat.

filebeat.prospectors:
- input_type: log
  paths: C:\Users\Sagar\Desktop\elastic_test4\data\log\*.log

output.logstash:
  hosts: ["10.101.00.11:5043"]

you must have to install 'ingest-geoip' filter plugin into elastic search. if you are not using any external database or services.

you can install elastic plugin using below command

elasticsearch-plugin install ingest-geoip

I'm not sure about your elastic instance because it's by-default listen 9200 port instead of 80 port.

You have to change your configuration script of logstash. something below like this.

input {
    beats {
        host => "10.101.00.11"
        port => "5044"
    }
}

filter {
    grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
    geoip { source => "clientip" }
}

output {
    elasticsearch {
        #hosts => ["elastic-instance-1.es.amazonaws.com:80"]
        hosts => ["elastic-instance-1.es.amazonaws.com:9200"]
        index => "apache-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}

after applying these configuration your output will look like.

{
  "_index": "apache-2017.09.21",
  "_type": "log",
  "_id": "AV6kqsr3A-YOTHfOm2US",
  "_version": 1,
  "_score": null,
  "_source": {
    "request": "/apps/cart.jsp?appID=9421",
    "agent": "\"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"",
    "geoip": {
      "city_name": "Beijing",
      "timezone": "Asia/Shanghai",
      "ip": "106.121.102.198",
      "latitude": 39.9289,
      "country_name": "China",
      "country_code2": "CN",
      "continent_code": "AS",
      "country_code3": "CN",
      "region_name": "Beijing",
      "location": {
        "lon": 116.3883,
        "lat": 39.9289
      },
      "region_code": "11",
      "longitude": 116.3883
    },
    "offset": 11050275,
    "auth": "-",
    "ident": "-",
    "input_type": "log",
    "verb": "POST",
    "source": "C:\\Users\\admin\\Desktop\\experiment\\Elastic\\access_log_20170915-005134.log",
    "message": "106.121.102.198 - - [19/Dec/2017:05:54:29 -0700] \"POST /apps/cart.jsp?appID=9421 HTTP/1.0\" 200 4984 \"http://cross.com/login/\" \"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "referrer": "\"http://cross.com/login/\"",
    "@timestamp": "2017-09-21T13:39:55.047Z",
    "response": "200",
    "bytes": "4984",
    "clientip": "106.121.102.198",
    "@version": "1",
    "beat": {
      "hostname": "DESKTOP-16QDF02",
      "name": "DESKTOP-16QDF02",
      "version": "5.5.2"
    },
    "host": "DESKTOP-16QDF02",
    "httpversion": "1.0",
    "timestamp": "19/Dec/2017:05:54:29 -0700"
  },
  "fields": {
    "@timestamp": [
      1506001195047
    ]
  },
  "sort": [
    1506001195047
  ]
}

I hope this is the solution you are looking for..

Answer 3

You may have to make sure the apache log is in the correct pattern:

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

For the pattern of the grok match, you can check the details on https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns.

Beside that, you might have a look on https://www.ip2location.com/tutorials/how-to-use-ip2location-filter-plugin-with-elk too.