Hadoop 2.9.2, Spark 2.4.0 access AWS s3a bucket

Question

It\'s been a couple of days but I could not download from public Amazon Bucket using Spark :(

Here is spark-shell command:

spark-shell           


        

Answer 1

I use spark 2.4.5 and this is what I did and it worked for me. I am able to connect to AWS s3 from Spark in my local.

(1) Download spark 2.4.5 from here:https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-without-hadoop-scala-2.12.tgz. This spark does not have hadoop in it.
(2) Download hadoop. https://archive.apache.org/dist/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
(3) Update .bash_profile
SPARK_HOME =  #example /home/spark-2.4.5/spark-2.4.5-bin-without-hadoop-scala-2.12
PATH=$SPARK_HOME/bin
(4) Add Hadoop in spark env
Copy spark-env.sh.template as spark-env.sh
add export SPARK_DIST_CLASSPATH=$( classpath)
here  is path to your hadoop /home/hadoop-3.2.1/bin/hadoop