2
votes

I am attempting to create GeoIP data using an ELK stack, which can be visualized in Kibana.

I have recently installed an ELK stack (Elastic Search, Logstash, and Kibana) on a virtual instance of Ubuntu Server 14.04. I am using Bro to capture logs.

Everything to do with capturing the logs, parsing them, and viewing them in Kibana is working great, except for GeoIP (one of the most interesting features!).

The GeoIP portion of my logstash config file looks like this;

geoip {
  add_tag => [ "geoip" ]
  database => "/etc/logstash/GeoLiteCity.dat" 
  source => "id.orig_h"
  target => "geoip"
  add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}"]
  add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
  convert => [ "[geoip][coordinates]", "float" ]
}

I had gotten that part of the filter from the following guide to setup GeoIP with Kibana. I have also seen that filter used in a few other places. I have attempted to simplify the filter (just source, target, database), but no luck.

When I do a curl request for my index, especially with the add_tag ["geoip"] it returns blank data;

"geoip" : {
        "dynamic" : "true",
        "properties" : {
          "location" : {
            "type" : "geo_point"
          }
        }

Finally, here is some scrubbed data that I have taken directly from Kibana, in JSON format;

{
  "_index": "logstash-2015.11.12",
  "_type": "bro-conn_log",
  "_id": "*****",
  "_score": null,
  "_source": {
    "message": [
      "*****"
    ],
    "@version": "1",
    "@timestamp": "2015-11-12T13:43:16.205Z",
    "host": "elk",
    "path": "/nsm/bro/logs/current/conn.log",
    "type": "bro-conn_log",
    "ts": "*****",
    "uid": "*****",
    "id.orig_h": "*****",
    "id.orig_p": *****,
    "id.resp_h": "*****",
    "id.resp_p": *****,
    "proto": "*****",
    "service": "*****",
    "duration": *****,
    "orig_bytes": *****,
    "resp_bytes": *****,
    "conn_state": "*****",
    "local_orig": "*****",
    "missed_bytes": *****,
    "history": "*****",
    "orig_pkts": *****,
    "orig_ip_bytes": *****,
    "resp_pkts": *****,
    "resp_ip_bytes": *****,
    "tunnel_parents": "*****",
    "column21": "(empty)",
    "conn_state_full": "*****"
  },
  "fields": {
    "@timestamp": [
      1447335796205
    ]
  },
  "sort": [
    1447335796205
  ]
}

To summarize: I am attempting to get GeoIP data working with an ELK stack. Despite following guides describing how to do exactly that, the GeoIP field will not display in Kibana. Any advice would be GREATLY appreciated.

1

1 Answers

3
votes

Very silly solution. The IP addresses that I was looking at were all internal. I assumed (incorrectly) that it would just generate empty GeoIP data for any un-resolvable IP addresses. But, as the geoIP documentation states:

Starting with version 1.3.0 of Logstash, a [geoip][location] field is created if the GeoIP lookup returns a latitude and longitude.

So without the Long and Lat a GeoIP field is never created. This has been confirmed by moving the machine to a more open network, and immediately seeing the GeoIP tag with the same filter above.