Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

Question

I recently upgraded to Elasticsearch version 6.1.1 and now I can\'t bulk index documents from a JSON file. When I do it inline, it works fine. Here are the contents of the d

Answer 1

For me the issue was only due to wrong file name. I have used customer_full.json in command where as the file was named customer_full in my file system (without the extention).

So in my case : curl -H "Content-Type: application/x-ndjson" -XPOST 'http://localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"customer_full"

command worked for me.

Answer 2

I was struggling with this for a hot minute ... Mine was caused by a space in my curl request between the --data and the -binary and gave the same error - "must end with new line [\n]}" ...

so double-check that in the curl req it's --data-binary not --data - binary

Answer 3

I ran into the same issue and spent hours adding and removing newlines before somebody pointed out I mis-typed the file name... So note that curl will throw the same error if the file is not actually present, making this super-confusing.

Answer 4

Press Enter end of the line inside the JSON file and run the command again.

curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"generated.json"

Answer 5

I had the same problem running on Windows 10, using ElasticSearch 7.5.1.

I tried all the answers; none of them worked. I was certain I had a newline at the end of my file.

To get it to work, I had to ensure the file I was uploading was using UNIX end-of-line characters (0A only, no 0D), and also the encoding had to be UTF-8.

Using Notepad++, you can edit the metadata of the file.

Finally some good news:

Answer 6

I faced the similar issue on windows using elastic 7.9.1 when i used below CURL command.

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/accounts/docs/_bulk?filter_path=items.*.error --data-binary  "@textoutES.json"  >> erroredAtES.json"

I tried to manually add Newline at the end of file but did not work.

I have created my JSON by extracting data from mysql database like below to make sure my records should end with LINE FEED and CARRIAGE RETURN.

Then, it is working for me.

SELECT CONCAT('{"index":{"_id":"',id,'"}}\r\n',request_data,'\r\n') reqestData FROM cards 

More Importantly you End-of-File should have a carriage-return and Line-Feed (CRLF)if you are using windows. Also if any line in JSON contains a CR but no LF then you will get parsing exception "Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@2d5ed2ca"

Windows CRLF and EOF