load a local file to spark using sc.textFile()

Question

Question

How to load a file from the local file system to Spark using sc.textFile? Do I need to change any -env variables? Also when I tried the same on my windows

Answer 1

I checked all the dependencies and the environment variables again. The actual path "file:///home/..../.. .txt" would fetch the data from the local file system as the hadoop env.sh file has its default file system set to fs.defaultFs. If we leave the Spark-env.sh to its defaults without any change it takes the local file system when it encounters "file://..." and the hdfs when the path is "hdfs://.." If you specifically need any file system export HADOOP_CONF_DIR to the spark-env.sh And it would support any file system supported by Hadoop. This was my observation. Any corrections or suggestions accepted. Thank you

Answer 2

This error happens when you run spark in a cluster. When you submit a job to spark cluster the cluster manager(YARN or Mesos or any) will submit it to worker node. When the worker node trying to find the path of the file we need to load into spark it fails because the worker doesn't have such file. So try running spark-shell in local mode and try again,

\bin\spark-shell --master local

sc.textFile("file:///C:/Users/swaapnika/Desktop/to do list")

let me know if this helps.

Answer 3

Try changing

val inputFile = sc.textFile("file///C:/Users/swaapnika/Desktop/to do list")

to this:

val inputFile = sc.textFile("file:///Users/swaapnika/Desktop/to do list")

I'm also fairly new to hadoop and spark, but from what I gather, when running spark locally on Windows, the string file:/// when passed to sc.textFile already refers to C:\.

Answer 4

The file path you have defined is incorrect.

Try changing

sc.textFile("file///C:/Users/swaapnika/Desktop/to do list")

to

sc.textFile("file://C:/Users/swaapnika/Desktop/to do list")

or

sc.textFile("C:/Users/swaapnika/Desktop/to do list")