ElasticSearch Logstash upload null value for date

Question

I am trying to bulk upload documents into elasticsearch(6.3) using logstash (6.3.1). One of the fields in the document in date. Below is my mapping type `

{
  "mappings": {
      "users" : {
        "properties" : {
            "name" : { "type" : "text" },
            "location" : { "type" : "geo_point"},
            "date": {
                "type": "date",
                "format": "yyyy-MM-dd"
            }

   }
  }
}

If there is no date available then I set it as null.

Ex csv upload doc

John, Seattle, null

During the upload, logstash reads the value for date as "null" (null with quotes) and the upload to elasticsearch fails because its doesn't accept "null" as a valid input to date. My question is how to let elasticsearch accept null values (null without quotes) for date when uploading from logstash?

Downvoting because this question can be awnsered with a quick search in google. In the documentation , there is a full explanation about null handling elastic.co/guide/en/elasticsearch/reference/current/… — Yeikel
This is not what I want. I clearly mentioned that when uploading from logstash, I want to replace null with quotes to without quotes — Newbie
I don't think null is a supported value in logstash, you might to remove the date field. You can do it like this: if [date] =~ /null/ { mutate { remove_field => [ "date" ] } } — baudsp
If you read the article you will see that null values are not supported and what are the ways to mediate in case of null — Yeikel

AKS AKS · Accepted Answer · 2020-05-11T21:46:17

Don't set it to null (as a string in csv file data). Set it to nothing/no value as all (i.e. for any such field where the value is not present, your csv file can have ,,, blank values.

In your case, for ex: Set it like this:

John,Seattle,
Shenzi,Denver,2020-05-11

For the first row, when logstash will read csv file (ex: if you had input { stdin{} } in logstash's conf file then you'll cat somefile.csv | logstash -f <logstash.conf>) it'll treat the 3rd column value as '' (turning it into an ES index as a key=val pair like: date: null for free)

ElasticSearch Logstash upload null value for date

1 Answers