问题
Does
org.apache.spark.sqlDataFrameReadercsv(path: String)
have an option for skipping blank lines? In particular, a blank line as the last line?
回答1:
You could try setting mode to "DROPMALFORMED" as in:
val df = sqlContext.read.format("com.databricks.spark.csv").option("mode", "DROPMALFORMED")...
In Python:
df = sqlContext.read.format('com.databricks.spark.csv').options(mode = "DROPMALFORMED")...
Which according to the documentation:
"...drops lines which have fewer or more tokens than expected."
来源：https://stackoverflow.com/questions/43476254/dataframereadercsvpath-string-option-for-skipping-blank-lines