Amazon s3a returns 400 Bad Request with Spark

Question

For checkout purpose I try to set up an Amazon S3 bucket as checkpoint file.

val checkpointDir = \"s3a://bucket-name/checkpoint.txt\"
val sc = new SparkConte         


        

Answer 1

I was facing the same issue when running spark locally, for me reason was SIGV4 was not getting set, this code helped me:

import com.amazonaws.SDKGlobalConfiguration
System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true")