Spark is inventing his own AWS secretKey

Question

I\'m trying to read a s3 bucket from Spark and up until today Spark always complain that the request return 403

hadoopConf = spark_context._jsc.hadoopConfigu         


        

Answer 1

I ran into a similar issue. Requests that were using valid AWS credentials returned a 403 Forbidden, but only on certain machines. Eventually I found out that the system time on those particular machines were 10 minutes behind. Synchronizing the system clock solved the problem.

Hope this helps!

Answer 2

It is very intriguing this random passkey. Maybe AWS SDK is getting the password from OS environment.

In hadoop 2.8, the default AWS provider chain shows the following list of providers:

BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider

Order, of course, matters! the AWSCredentialProviderChain, get the first keys from the first provider that provides that information.

            if (credentials.getAWSAccessKeyId() != null &&
                credentials.getAWSSecretKey() != null) {
                log.debug("Loading credentials from " + provider.toString());
                lastUsedProvider = provider;
                return credentials;
            } 

See the code in "GrepCode for AWSCredentialProviderChain".

I face similar problem using profile credentials. SDK was ignoring the credentials inside ~/.aws/credentials (as good practice, I encourage you to not store credentials inside the program in any way).

My solution...

Set the credentials provider to use ProfileCredentialsProvider

sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.eu-central-1.amazonaws.com") # yes, I am using central eu server.
sc._jsc.hadoopConfiguration().set('fs.s3a.aws.credentials.provider', 'com.amazonaws.auth.profile.ProfileCredentialsProvider')

Answer 3

(updated as my original one was downvoted as clearly considered unacceptable)

The AWS auth protocol doesn't send your secret over the wire. It signs the message. That's why what you see isn't what you passed in.

For further information, please reread.