s3

Cannot access S3 bucket with Hadoop

匿名 (未验证) 提交于 2019-12-03 10:24:21
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I'm trying to access my S3 bucket with Hadoop (2.7.3) and I'm getting the following ubuntu@AWS:~/Prototype/hadoop$ ubuntu@AWS:~/Prototype/hadoop$ bin/hadoop fs -ls s3://[bucket]/ 17/03/24 15:33:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable -ls: Fatal internal error com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 1FA2318A386330C0, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended

AWS S3 upload without access and secret key in Java

匿名 (未验证) 提交于 2019-12-03 09:06:55
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I want to upload a file to S3 without using my access and secret key from AWS server. AWS keys should be taken as default. However running the below command in server I can access it without providing any access and secret keys. aws s3 cp somefile.txt s3://somebucket/ From java code its not accessible since it was unable to load credentials. Below is my code. AmazonS3 s3client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain()); 回答1: You can use the below Java code to get the s3client instance when you are trying to connect to S3

Why is my terminal returning this s3 error?

匿名 (未验证) 提交于 2019-12-03 09:05:37
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: Here's the error I keep receiving: A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied I've triple-checked my credentials and googled this error to my wits' end. I edited my bucket policy to add an s3:ListBucket action, but to no avail. When I do so, it just returns a similar message: A client error (AccessDenied) occurred when calling the ListBuckets operation: Access Denied Thoughts? This is also my first time creating an s3 bucket so it's quite possible I missed some important step. I have triple

How to get the file diff between two S3 buckets?

匿名 (未验证) 提交于 2019-12-03 08:59:04
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: So I have an S3 bucket of videos (several hundred), upon which I used ElasticTranscoder to transcode everything into a second, optimised bucket. However, when I inspect my second bucket, there are 40-50 less objects, but I cannot figure out what they are (the directory structure is deeply nested etc). How can I get the file diff of two buckets using aws s3api list-objects ? Perhaps there are files in the bucket which are not videos, which I somehow didn't know about. 回答1: Using Display only filenames : aws s3 ls s3://bucket-1 --recursive |

Amazon S3 Upload error PermanentRedirectException

匿名 (未验证) 提交于 2019-12-03 08:57:35
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I've installed the Amazon SDK for PHP using composer. I copied the example code and implemented into my system, but when I attempt to upload a file, I'm getting the following (i've made the paths generic for security): Fatal error: Uncaught exception 'Aws\S3\Exception\PermanentRedirectException' with message 'Encountered a permanent redirect while requesting https://s3.amazonaws.com/mybucket/path/to/image.jpg . Are you sure you are using the correct region for this bucket?' in /path/to/vendor/aws/aws-sdk-php/src/S3

AWS EC2 IAM role access denied on S3

匿名 (未验证) 提交于 2019-12-03 08:56:10
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I have launched an EC2 instance with IAM role "webapp". role is attached and i can confirm it using curl http://169.254.169.254/latest/meta-data/iam/security-credentials/webapp { "Code" : "Success", "LastUpdated" : "2016-01-04T06:44:50Z", "Type" : "AWS-HMAC", "AccessKeyId" : "xxx", "SecretAccessKey" : "xxx", "Token" : "xxx", "Expiration" : "2016-01-04T12:46:27Z" } webapp Role has an attached policy for S3 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": "*" } ] } But I am unable to access objects

AWS S3 Pre-signed URL via CLI

匿名 (未验证) 提交于 2019-12-03 08:54:24
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: Is there a way to create pre-signed url for objects in S3 bucket using AWS CLI? I know that could be done using SDK, but is it possible with CLI? I found this on one of the AWS docs, but can't complete the command: s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset> Any help? 回答1: Did you try aws s3 presign ? Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. For sigv4 requests the region needs to be configured explicitly. This

Trying to set up Amazon's S3 bucket: 403 Forbidden error &amp; setting permissions

匿名 (未验证) 提交于 2019-12-03 08:51:18
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 由 翻译 强力驱动 问题: I'm following Hartl's railstutorial.org and have arrived at 11.4.4: Image upload in production . What I've done: Signed up for Amazon Web Services In Amazon Identity and Access Management I created a user. The user was created succesfully. In Amazon S3 I created a new bucket. Set permissions on the new bucket: Permissions: The tutorial instructs to "grant read and write permission to the user created in the previous step". However, under 'Permissons' for the bucket the new user's name is not mentioned. I could only choose between

Writing an IAM policy and CORS configuration for Amazon S3

匿名 (未验证) 提交于 2019-12-03 08:48:34
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am very new to all this but have been able to get an avatar/image uploader to work in my Rails application. A user can upload a new avatar to my S3 bucket and the avatar shows within the web application. To this end, I've had to grant "AmazonS3FullAccess" policy to the user. That seems like a bit too much, since the user from the application only needs write (upload his avatar) and read (show the avatar on the web page) permission. Would you agree that it is therefore better to write a custom policy rather than use AmazonS3FullAccess? If