s3

AWS S3 website versioning

匿名 (未验证) 提交于 2019-12-03 01:38:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am running a website on AWS S3 bucket. I have to update the website once in a while. At the moment, when I do the deployment I just copy the built files to my bucket and override existing ones. Is there a way to do some versioning on these deployments? I know there is a built-in versioning S3, but it is only for individual files I think. The best option would be that every deployment is tagged with git commit-id and I could rollback to a particular commit-id if needed. Any ideas? Already tried to name directories with commit-id -prefix,

AWS Glue Crawler Not Creating Table

匿名 (未验证) 提交于 2019-12-03 01:33:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 由 翻译 强力驱动 问题: I have a crawler I created in AWS Glue that does not create a table in the Data Catalog after it successfully completes. The crawler takes roughly 20 seconds to run and the logs show it successfully completed. CloudWatch log shows: Benchmark: Running Start Crawl for Crawler Benchmark: Classification Complete, writing results to DB Benchmark: Finished writing to Catalog Benchmark: Crawler has finished running and is in ready state I am at a loss as to why the tables in the data catalog are not being created. AWS Docs are not of much

AWS S3 Bucket Permissions - Access Denied

匿名 (未验证) 提交于 2019-12-03 01:33:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am trying to give myself permission to download existing files in an S3 bucket. I've modified the Bucket Policy, as follows: { "Sid": "someSID", "Action": "s3:*", "Effect": "Allow", "Resource": "arn:aws:s3:::bucketname/AWSLogs/123123123123/*", "Principal": { "AWS": [ "arn:aws:iam::123123123123:user/myuid" ] } } My understanding is that addition to the policy should give me full rights to "bucketname" for my account "myuid", including all files that are already in that bucket. However, I'm still getting Access Denied errors when I try to

setting up s3 for logs in airflow

匿名 (未验证) 提交于 2019-12-03 01:27:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am using docker-compose to set up a scalable airflow cluster. I based my approach off of this Dockerfile https://hub.docker.com/r/puckel/docker-airflow/ My problem is getting the logs set up to write/read from s3. When a dag has completed I get an error like this *** Log file isn't local. *** Fetching here: http://ea43d4d49f35:8793/log/xxxxxxx/2017-06-26T11:00:00 *** Failed to fetch log file from worker. *** Reading remote logs... Could not read logs from s3://buckets/xxxxxxx/airflow/logs/xxxxxxx/2017-06- 26T11:00:00 I set up a new section

S3 module for downloading files is not working in ansible

匿名 (未验证) 提交于 2019-12-03 01:03:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: This is the ansible code written for downloading files from S3 bucket "artefact-test". - name: Download customization artifacts from S3 s3: bucket: "artefact-test" object: "cust/gitbranching.txt" dest: "/home/ubuntu/" mode: get region: "{{ s3_region }}" profile: "{{ s3_profile }}" I have set the boto profile and aws profile too. I get different errors which i dont think are valid like - failed: [127.0.0.1] => {"failed": true, "parsed": false} Traceback (most recent call last): File "/home/dmittal/.ansible/tmp/ansible-tmp-1462436903.77

AWS C++ S3 SDK PutObjectRequest unable to connect to endpoint

匿名 (未验证) 提交于 2019-12-03 00:59:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: In working with the AWS C++ SDK I ran into an issue where trying to execute a PutObjectRequest complains that it is "unable to connect to endpoint" when uploaded more than ~400KB. Aws::Client::ClientConfiguration clientConfig; clientConfig.scheme = Aws::Http::Scheme::HTTPS; clientConfig.region = Aws::Region::US_EAST_1; Aws::S3::S3Client s3Client(clientConfig); Aws::S3::Model::PutObjectRequest putObjectRequest; putObjectRequest.SetBucket("mybucket"); putObjectRequest.SetKey("mykey"); typedef boost::iostreams::basic_array_source<char> Device;

How do I translate an AWS S3 url into a bucket name for boto?

匿名 (未验证) 提交于 2019-12-03 00:59:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I'm trying to access the http://s3.amazonaws.com/commoncrawl/parse-output/segment/ bucket with boto. I can't figure out how to translate this into a name for boto.s3.bucket.Bucket(). This is the gist of what I'm going for: s3 = boto.connect_s3() cc = boto.s3.bucket.Bucket(connection=s3, name='commoncrawl/parse-output/segment') requester = {'x-amz-request-payer':'requester'} contents = cc.list(headers=requester) for i,item in enumerate(contents): print item.__repr__() I get "boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ...

AWS s3 download all file inside a specific folder - Using PHP SDK

匿名 (未验证) 提交于 2019-12-03 00:59:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: Can anyone help me for this one..?? I want to download all files from a folder which is inside my bucket's folder, to my computer's directory with the same name. Let's say there is a bucket name "ABC" a folder is there inside it, is "DEF".. In which folder there are multiple files available.. Now I want to download it into my project folder "/opt/lampp/htdocs/porject/files/download/" here "DEF" folder is also available.. So, anyone can help me, and give me the code for this..? Thanks in advance.. ============= ERROR : Fatal error: Uncaught

PHP Amazon SDK, S3 Bucket Access Denied

匿名 (未验证) 提交于 2019-12-03 00:56:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I try for the first time to use the PHP AWS SDK ("aws/aws-sdk-php": "^3.19") to use S3. I created a bucket : 'myfirstbucket-jeremyc' I created a policy : { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::myfirstbucket-jeremyc/*" ] } ] } I applied the policy to a group and then created a user 's3-myfirstbucket-jeremyc' in this group. My PHP code is : <?php use Aws\S3\S3Client; use Aws\S3\Exception\S3Exception; error_reporting(E_ALL);

How to resolve AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null) using java

匿名 (未验证) 提交于 2019-12-03 00:56:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: Stack trace Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null), S3 Extended Request ID: null at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com