Problems bulk insert elasticsearch curl powershell

Question

I'm new to Elasticsearch and I'm following the tutorial loading sample data: https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html When I try to bulk insert the data in powershell 'curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/_bulk?pretty' --data-binary "@shakespeare.json"'

I get the following error

PS C:\Development\elasticsearch-5.4.1\importdata> curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/_bulk?pretty' --data-binary "@shakespeare.json" Invoke-WebRequest : Cannot bind parameter 'Headers'. Cannot convert the "Content-Type: application/x-ndjson" value of type "System.St ring" to type "System.Collections.IDictionary". At line:1 char:9 + curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/s ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest], ParameterBindingException + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

I've tried several things, read the bulk api documentation but with no result

Here're some examples of the content of the json file:

{"index":{"_index":"shakespeare","_type":"act","_id":0}} {"line_id":1,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"} {"index":{"_index":"shakespeare","_type":"scene","_id":1}} {"line_id":2,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"SCENE I. London. The palace."} {"index":{"_index":"shakespeare","_type":"line","_id":2}} {"line_id":3,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"Enter KING HENRY, LORD JOHN OF LANCASTER, the EARL of WESTMORELAND, SIR WALTER BLUNT, and others"}

The structure of the json data is different when using bulk injection, you basically need to add the index/type/id as part of the json: there is more info here elastic.co/guide/en/elasticsearch/reference/5.4/docs-bulk.html — Micky Balladelli
also you are using unix/curl litterally in powershell/Invoke-WebRequest, you should use Invoke-RestMethod instead imo. — Micky Balladelli
finally the URI should be localhost:9200/_bulk..these comments wont solve it all since I didn't see the sharespeare.json file, but it should get you closer. — Micky Balladelli
I added some example lines of the shakespeare.json file. Everything seems correct. I tried the Invoike-RestMethod but no result — Werner Kellens
Does anybody know why I get this exception 'Invoke-WebRequest : Cannot bind parameter 'Headers'. Cannot convert the "Content-Type: application/x-ndjson" value of type "System.St ring" to type "System.Collections.IDictionary"' — Werner Kellens

Werner Kellens Werner Kellens · Accepted Answer · 2017-06-05T17:53:43

I found the solution. Because I used curl on windows and with powershell, I had to translate

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/_bulk?pretty' --data-binary @shakespeare.json

to

PS C:\Development\elasticsearch-5.4.1\importdata> $postParams = @"
>> {"index":{"_index":"shakespeare","_type":"act","_id":0}}
>> {"line_id":1,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}
>>
>> "@

PS C:\Development\elasticsearch-5.4.1\importdata> curl -H @{"Content-Type" = "application/x-ndjson"} -Method POST 'http://localhost:92
00/shakespeare/_bulk?pretty' -body $postParams

When I use the demo file, the bulk insert didn't succeeded but that's another problem

PS C:\Development\elasticsearch-5.4.1\importdata> $postParams = Get-Content "shakespeare.json"

Problems bulk insert elasticsearch curl powershell

1 Answers