logstash remove_field not working in order to upload csv to elasticsearch

Question

I'm using elasticsearch, kibana and logstash 6.0.1.

I wish to upload csv data to elasticsearch by logstash and removing fields (path, @timestamp, @version, host and message). I'm showing logstash.conf and emp.csv files below. The upload will work if I don't use the remove_field instruction but I need to. Furthermore, the index was not created.

logstash.conf:

input {

  file {
      path => "e:\emp.csv"
      start_position => "beginning"
  }
}
filter {
  csv {
      separator => ","
      columns => ["code","color"]
      remove_field => ["path", "@timestamp", "@version", "host", "message"]
  }

  mutate {convert => ["code", "string"]}
  mutate {convert => ["color", "string"]}

}
output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "emp5"
    user => "elastic"
    password => "password"
  }
  stdout {}
}

emp.csv:

1,blue
2,red

What is missing in this case?

Kamil Witkowski Kamil Witkowski · Accepted Answer · 2019-12-10T11:54:33

In your csv file the data is not available that you are trying to delete.

Instead try this to delete for example path and host field:

(...)
filter {
  csv {
      separator => ","
      columns => ["code","color"]
  }
    mutate {
      remove_field => ["path", "host"]
    }
(...)

And for information, if field path and/or host doesn't exist, there's no problem. The plugin will remove field if field exists, and just do nothing if field does not exist.

Edit: I have tested it on fresh elastic stack:

You can delete index with:

curl -X DELETE "localhost:9200/emp5"

Also note that in your current config logstash will read the file only once. You can change that behaviour by adding sincedb_path => "/dev/null" or in Windows case: sincedb_path => "NUL" inside:

input {
    file {
           (...) # here
         }

section.

Then after logstash work verify result with:

curl -X GET "localhost:9200/emp5?pretty"
{
  "emp5" : {
    "aliases" : { },
    "mappings" : {
      "doc" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "code" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "color" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "blocks" : {
          "read_only_allow_delete" : "true"
        },
        "provided_name" : "emp5",
        "creation_date" : "1576099826712",
        "number_of_replicas" : "1",
        "uuid" : "reXYzqPgQryYcASoov9l5A",
        "version" : {
          "created" : "6080599"
        }
      }
    }
  }
}

As you can see there is no host and path field.

logstash remove_field not working in order to upload csv to elasticsearch

1 Answers