how to use bulk API to insert null value in date format to elasticsearch

Question

in elasticsearch,I have a date field setup in my mapping like this:

"create_time" : {
 "type" : "date",
 "null_value" : "NULL"
 },

i use pandas to process my data,it like this:

    create_date
0       NULL
1       2019-09-23 14:32:32.217377

and in python i use bulk API,the method is:

def gendata(data, index):
    INDEX = index
    TYPE = "doc"
    for record in data:
        yield {
            "_index": INDEX,
            "_type": TYPE,
            "_source": record,
        }
bulk(client,gendata(data.to_dict(orient="records"),index="test"))

but i get BulkIndexError:('1 document(s) failed to index.', [{'index': {'_index': 'ics_asset', '_type': 'doc', '_id': '85vaXG0Be27aFy3eNkCZ', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'mapper [create_date] of different type, current_type [date], merged_type [text]'}, 'data': {'create_date': 'null'}}}]).

how can i use pandas bulk right "null" value with date format to elasticsearch.

thanks

frankie567 frankie567 · Accepted Answer · 2019-09-23T07:01:51

The documentation about null_value clearly states:

The null_value needs to be the same datatype as the field. For instance, a long field cannot have a string null_value.

This means you cannot assign the string NULL as null_value for a date field.

This null_value is only useful if you want to explicitly query documents with no value for a particular field. Maybe you can assign a date like 1900-01-01 so that you can query them with that particular date.

Otherwise, you can drop the null_value option and you'll be perfectly able to index documents with a regular null value. The only downside is that you won't be able to retrieve documents by null create date.

how to use bulk API to insert null value in date format to elasticsearch

1 Answers