amazon-s3

Controlling where HTTP requests go in AWS

由 99封情书 提交于 2020-03-05 14:20:09
问题 If I have a static site on AWS S3 (and maybe using CloudFront) that's pretty cool, because it scales easily, and has zero-downtime deployments, because you're just updating static assets, and gets distributed to edge locations, woohoo! But, if I want to have a contact form, or process a stripe payment. I need to run some backend code. So, how do I tell AWS that for GETs to certain routes, use S3 (or CloudFront), but if there's a form submit, direct that to this little Lambda function over

Controlling where HTTP requests go in AWS

由 北战南征 提交于 2020-03-05 14:19:52
问题 If I have a static site on AWS S3 (and maybe using CloudFront) that's pretty cool, because it scales easily, and has zero-downtime deployments, because you're just updating static assets, and gets distributed to edge locations, woohoo! But, if I want to have a contact form, or process a stripe payment. I need to run some backend code. So, how do I tell AWS that for GETs to certain routes, use S3 (or CloudFront), but if there's a form submit, direct that to this little Lambda function over

Error trying to access AWS S3 using Pyspark

由 自古美人都是妖i 提交于 2020-03-05 03:38:09
问题 I am trying to access gzip files from AWS S3 using Spark. I have a very simple script below. I first started off with a IAM user with access permissions to the S3 bucket. Then I created an EC2 instance & installed Python & Spark. I setup the spark.properties file as below. I only copied the jar files, didn't bother to go through the entire Hadoop installation. Then I realized I have to create an IAM role for EC2 instances to access S3. So, I created an IAM role, attached an access policy and

While writing to S3, why I get FileNotFoundException

由 别等时光非礼了梦想. 提交于 2020-03-05 00:22:42
问题 I'm using Spark-SQL-2.3.1, Kafka, Java 8 in my project, and would like to use AWS-S3 as savage storage. I am writing/storing the consumed data from Kafka topic into S3 bucket as below: ds.writeStream() .format("parquet") .option("path", parquetFileName) .option("mergeSchema", true) .outputMode("append") .partitionBy("company_id") .option("checkpointLocation", checkPtLocation) .trigger(Trigger.ProcessingTime("25 seconds")) .start(); But while writing I am getting a FileNotFoundException Caused

AWS STS to list buckets gives access denied

由 大城市里の小女人 提交于 2020-03-04 19:36:09
问题 I have a bucket with empty bucket policy, block public access turned ON (ACLs and Bucket) and trying to list buckets using IAM policy tied to user using STS AssumeRole with following attached policy. { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetObject", "s3:GetBucket*", "s3:ListBucket*", "s3:ListAllMyBuckets" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::my-test-bucket/*" ] } ] } The assumed role credentials are used during the STS session in python (boto3) s3c = boto3

Retrieve data from Amazon s3

由 别说谁变了你拦得住时间么 提交于 2020-03-04 08:01:52
问题 I am trying to retrieve the URL of a picture from Amazon s3. When I run the script below, I get an error: Missing required Key in params This is what I have so far: <html xmlns="http://www.w3.org/1999/xhtml"> <head> <script src="https://sdk.amazonaws.com/js/aws-sdk-2.0.16.min.js"></script> <script type="text/javascript"> function test1(){ AWS.config.update({ accessKeyId: 'accesskey', secretAccessKey: 'secretKey' }); AWS.config.region = 'us-west-2'; var myAWS = new AWS.S3(); myAWS.getObject( {

Uploading CSV File to S3 using Rest Assured

由 北城以北 提交于 2020-03-04 05:04:33
问题 I am trying to replicate a postman put request where it is uploading a csv file on the S3 location given by a code. The url is like https://us-east-1-e9qpbo283.s3.amazonaws.com/bulk-bucket/a4894e7b-7e42-4fcc-9f84-e7n00db6d581/input/file with query params as {X-Amz-Date=20200226T113914Z, X-Amz-Algorithm=AWS4-HMAC-SHA256, X-Amz-Signature=<Some Signature>, X-Amz-SignedHeaders=content-type%3Bhost, X-Amz-Security-Token=<SOME TOKEN HERE>, X-Amz-Credential=ASIAV7AYOYCBQB4VDGD7%2F20200226%2Fus-east-1

How to filter s3 objects by last modified date with Boto3

由 ⅰ亾dé卋堺 提交于 2020-03-03 07:42:11
问题 Is there a way to filter s3 objects by last modified date in boto3? I've constructed a large text file list of all the contents in a bucket. Some time has passed and I'd like to list only objects that were added after the last time I looped through the entire bucket. I know I can use the Marker property to start from a certain object name,so I could give it the last object I processed in the text file but that does not guarantee a new object wasn't added before that object name. e.g. if the

Amazon s3a returns 400 Bad Request with Spark-redshift library

由 一世执手 提交于 2020-03-03 04:23:53
问题 I am facing java.io.IOException: s3n://bucket-name : 400 : Bad Request error while loading Redshift data through spark-redshift library: The Redshift cluster and the s3 bucket both are in mumbai region . Here is the full error stack: 2017-01-13 13:14:22 WARN TaskSetManager:66 - Lost task 0.0 in stage 0.0 (TID 0, master): java.io.IOException: s3n://bucket-name : 400 : Bad Request at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:453)

Amazon s3a returns 400 Bad Request with Spark-redshift library

由 假如想象 提交于 2020-03-03 04:22:24
问题 I am facing java.io.IOException: s3n://bucket-name : 400 : Bad Request error while loading Redshift data through spark-redshift library: The Redshift cluster and the s3 bucket both are in mumbai region . Here is the full error stack: 2017-01-13 13:14:22 WARN TaskSetManager:66 - Lost task 0.0 in stage 0.0 (TID 0, master): java.io.IOException: s3n://bucket-name : 400 : Bad Request at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:453)
订阅 amazon-s3