Dealing with “null” fields and add_field directive in logstash

Question

I have a json file with 100 records. Its structure is like this:

{
    "_app1": {
        "test": "test"
    },
    "location": {
        "longitude": 40.400000000000006,
        "latitude": 40.400000000000006,
        "country": "CH"
    },
    "timestamp": "2015-08-23"
}

I need to create some Geo visualizatons from Kibana so I have defined the geo_point type in the mappings file this way:

"location" : { "type" : "geo_point" }

Since the location field has to follow certain structure (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html) I have done the following in the logstash config file:

input {
  stdin {
    type => "json"
  }
}

filter {
   json {
        source => "message"
    }
   mutate {
        rename => [ "location", "newlocation" ]
        add_field => { "location" => "%{[newlocation][latitude]},%{[newlocation][longitude]}" }
    }

}

output {
    elasticsearch {
...
    }
}

With this config I get that most of the documents are indexed in elasticsearch, but (and here is the problem) if the longitude or the latitude is "null" the register is not indexed. So, for instance, this record:

{
    "_app1": {
        "test": "test"
    },
    "location": {
        "longitude": null,
        "latitude": 40.400000000000006,
        "country": "CH"
    },
    "timestamp": "2015-08-23"
}

would not be indexed in ES. My question is, how can I index all the registers in ES and create a new field for those whose latitude and longitude are different that null.

I have tried something like this:

...
filter {
   json {
        source => "message"
    }
   mutate {
        rename => [ "location", "newlocation" ]
    }
   if [newlocation][latitude] and [newlocation][longitude] {
     mutate {
        add_field => { "location" => "%{[newlocation][latitude]},%{[newlocation][longitude]}" }
     }
   }
}
..

but not work, any ideas?

Your sample JSON doesn't seem well-formed, i.e. it is not possible to have this {"_app1": "location": {.... Are you missing a pair of curly braces? If you run your JSON though jsonlint.com, you'll get an error. — Val

Alain Collins Alain Collins · Accepted Answer · 2015-09-25T01:01:09

Your config works for me in this test harness:

input {
    stdin {  }
}

filter {
  json {
        source => "message"
  }

  mutate {
        rename => [ "location", "newlocation" ]
  }

  if [newlocation][latitude] and [newlocation][longitude] {
     mutate {
        add_field => { "location" => "%{[newlocation][latitude]},%{[newlocation][longitude]}" }
     }
   }
}

output {
    stdout {
        codec => rubydebug
    }
}

Dealing with “null” fields and add_field directive in logstash

1 Answers