I have a forward slash in aws secret key.
When I try to connect to s3 sink
Caused by: java.lang.IllegalArgumentException: Invalid hostname in URI s3://xxxx:xxxx@jelogs/je.1359961366545
at org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:41)
When I encode forward slash with %2F, I get
The request signature we calculated does not match the signature you provided. Check your key and signing method.
How should I encode my secret key.
samthebest solution works, you just have to add "" surrounding the keys. Here how to use it:
hadoop distcp -Dfs.s3a.awsAccessKeyId="yourkey" -Dfs.s3a.awsSecretAccessKey="yoursecret" <your_hdfs_path> s3a://<your-bucket>
I end up creating a new secret key without forward slashes. This is a know issue and generating new key is only solution.
Use
-Dfs.s3n.awsAccessKeyId=<your-key> -Dfs.s3n.awsSecretAccessKey=<your-secret-key>
e.g.
hadoop distcp -Dfs.s3n.awsAccessKeyId=<your-key> -Dfs.s3n.awsSecretAccessKey=<your-secret-key> -<subsubcommand> <args>
or
hadoop fs -Dfs.s3n.awsAccessKeyId=<your-key> -Dfs.s3n.awsSecretAccessKey=<your-secret-key> -<subsubcommand> <args>
来源：https://stackoverflow.com/questions/14681938/invalid-hostname-error-when-connecting-to-s3-sink-when-using-secret-key-having-f