Elasticdump creats one document with 1000 fields when importing a json file in elasticsearch

Question

There are 1000 records fetch from Database and formatted in to JSON. I need to insert each one as a document in Elasticsearch and autocomplete_entities index. The input file looks like this

[
    {
        "id" : 1,
        "title" : "x"
    },
    {
        "id" : 2,
        "title" : "y"
    },
    ...
]

I have these settings too

PUT _settings
{
    "index.mapping.total_fields.limit" : 10000
}

and

put autocomplete_entities

Now when I run this code

elasticdump --bulk=true --input="PycharmProjects/untitled/v1.json" --output="http://localhost:9200/_doc" --output-index="autocomplete_entities" --type=data --transform="doc._source=Object.assign({},doc)"

It creates one document with 1000 fields such that each fields contains id, title, as follows

"took": 9,
"timed_out": false,
"_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
        {
            "_index": "autocomplete_entities",
            "_type": "autocomplete_entities",
            "_id": "voVp7XQBq56KEuWZ8JuQ",
            "_score": 1,
            "_source": {
                "0": {
                    "id": 37905,
                     "title": "x"
                },
                "1": {
                    "id": 44093,
                    "title": "y"
                } ...
            }
        }
    ]
}

How can I import 1000 records in elasticsearch as 1000 documents ?

user156327 user156327 · Accepted Answer · 2020-10-03T10:40:23

can you define mapping for your autocomplete_entities index as below

{
    "mappings": {
        "properties": {
            "data": {
                "properties": {
                    "id": {
                        "type": "integer"
                    },
                    "title": {
                        "type": "text"
                    }
                }
            }
        }
    }
}

And see if its getting indexed properly, this way it would prevent creating the new fields for all your documents, please note that data is the object type of elasticsearch and this is how data can be indexed in above index:

{
    "data": [
        {
            "id": 1,
            "title": "x"
        },
        {
            "id": 2,
            "title": "y"
        }
    ]
}

Elasticdump creats one document with 1000 fields when importing a json file in elasticsearch

1 Answers