Spark Scala read csv file using s3a

Question

I am trying to read a csv (native) file from an S3 bucket using a locally running Spark - Scala. I am able to read the file using the http protocol but I intend to use the s3a p

Answer 1

Anyone else struggling with this I had to update the version of hadoop-client

additionally the links below were quite helpful

• https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html

• https://disqus.com/by/cfeduke/?utm_source=reply&utm_medium=email&utm_content=comment_author

• http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

pom details below

    2.2.0
    2.8.0





    
    
        org.apache.spark
        spark-core_2.11
        ${spark.version}
    
    
        org.apache.spark
        spark-sql_2.11
        ${spark.version}
    
    
        org.apache.hadoop
        hadoop-client
        ${hadoop.version}
    
    
        org.apache.hadoop
        hadoop-aws
        ${hadoop.version}