Bulk request throws error in elasticsearch 6.1.1

Question

I recently upgraded to elasticsearch version 6.1.1 and now I can't bulk index documents from a json file. Wehn I do it inline, it works fine. Here are the contents of the document:

{"index" : {}}
{"name": "Carlson Barnes", "age": 34}
{"index":{}}
{"name": "Sheppard Stein","age": 39}
{"index":{}}
{"name": "Nixon Singleton","age": 36}
{"index":{}}
{"name": "Sharron Sosa","age": 33}
{"index":{}}
{"name": "Kendra Cabrera","age": 24}
{"index":{}}
{"name": "Young Robinson","age": 20}

When I run this command,

curl -XPUT 'localhost:9200/subscribers/ppl/_bulk?pretty' -H 'Content-Type: application/json' -d @customers_full.json

I get this error:

"error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "The bulk request must be terminated by a newline [\n]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "The bulk request must be terminated by a newline [\n]"
  },
  "status" : 400

It works fine if I send the data inline and in elasticsearch 5.x. I tried adding newlines as well as the newline character to the end of the file. Doesn't seem to work.

Answer 1

Add empty line at the end of the JSON file and save the file and then try to run the below command

curl -XPOST localhost:9200/subscribers/ppl/_bulk?pretty --data-binary @customers_full.json -H 'Content-Type: application/json'

I hope it works fine for you.

Answer 2

As the document says:

use the --data-binary flag instead of plain -d

-d doesn’t preserve newlines and do not format the json.

I faced this problem because of JSON formatting.

Answer 3

The error is pretty clear:

The bulk request must be terminated by a newline [\n]

So you simply need to add a newline at the end of your customers_full.json file and you'll be ok.

Answer 4

I ran into the same issue and spent hours adding and removing newlines before somebody pointed out I mis-typed the file name... So note that curl will throw the same error if the file is not actually present, making this super-confusing.

Answer 5

you just need to open json file and then go to the end of the file ( Ctrl+end) and then please Enter to break a new line.

Answer 6

You need to use --data-binary instead of -d in your curl request. Please see https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

Answer 7

Press Enter end of the line inside json file and run the command again .

curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"generated.json"

Answer 8

For anyone using postman to make requests to ElasticSearch

Just press enter to create an empty new line!

And voila, problem solved

Answer 9

I had a similar issue when working with Elasticsearch 7.3.

Here's how I solved it.

1. Locate the .json file, say products.json file.
2. Double click to open the .json file in your text editor.
3. Scroll to the end of the .json file and then press the ENTER key on your keyboard.
4. Close the .json file. This will create a new line at the end of .json file.
5. Go back to your terminal and run the command below.

N/B: For the command below, the .json file name is products.json which I am importing to http://localhost:9200/ecommerce/product

curl -H "Content-type: application/json" -XPOST "http://localhost:9200/ecommerce/product/_bulk?pretty" --data-binary "@products.json"

That's all.

I hope this helps

Answer 10

This worked in my local set-up.

curl -H "Content-type:application/json" -XPOST "http://localhost:9200/customer/personal/_bulk?pretty" --data-binary @"generated.json"

Answer 11

This worked for me:

curl -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/bank/_bulk?pretty&refresh" --data-binary "@C:\Program Files\Elastic\Elasticsearch\7.2.0\accounts.json"