Write to elasticsearch from spark is very slow
问题 I am processing a text file and writing transformed rows from a Spark application to elastic search as bellow input.write.format("org.elasticsearch.spark.sql") .mode(SaveMode.Append) .option("es.resource", "{date}/" + dir).save() This runs very slow and takes around 8 minutes to write 287.9 MB / 1513789 records. How can I tune spark and elasticsearch settings to make it faster given that network latency will always be there. I am using spark in local mode and have 16 cores and 64GB RAM. My