Spark to read a big file as inputstream

问题

I know spark built in method can have partition and read huge chunk of file and distributed as rdd using textfile. However, i am reading this in a customized encrytped filessytem which spark does not support by nature. One way i can think of is to read an inputstream instead and loads multiple lines and distributed to executor. Keep reading until all file is loaded. So no executor will blow up due to out of memory error. Is that possible to do this in spark?

回答1:

you can try lines.take(n) for different n to find the limit of your cluster.
or

spark.readStream.option("sep", ";").csv("filepath.csv")

来源：https://stackoverflow.com/questions/43023884/spark-to-read-a-big-file-as-inputstream