amazon-s3

问题 I want to list only the objects in a bucket that aren't buckets themselves. Is there a way of doing this short of parsing out the results of ListBucket? 回答1: objects in a bucket that aren't buckets themselves Buckets can't contain other buckets. Do you mean folders? S3 doesn't have a concept of folders either. You can have 100 buckets per S3 account and each bucket can contain an unlimited number of objects/files. If you name your files with / 's in the filename, the AWS GUI tools (eg AWS

问题 I want to list only the objects in a bucket that aren't buckets themselves. Is there a way of doing this short of parsing out the results of ListBucket? 回答1: objects in a bucket that aren't buckets themselves Buckets can't contain other buckets. Do you mean folders? S3 doesn't have a concept of folders either. You can have 100 buckets per S3 account and each bucket can contain an unlimited number of objects/files. If you name your files with / 's in the filename, the AWS GUI tools (eg AWS

问题 I'm having a really hard time setting up my bucket policy, it looks like my bucket policy only applies to some objects in my bucket. What I want is pretty simple: I store video files in the bucket and I want them to be exclusively downloadable from my webiste. My approach is to block everything by default, and then add allow rules: Give full rights to root and Alice user. Give public access to files in my bucket from only specific referers (my websites). Note: I manually made all the objects

问题 I have two different sources in my CodePipeline, ECR and S3. My deployment pipeline uploads a zip to S3, and then an image to ECR. I need CodePipeline to detect just the ECR commit, which happens last, and then trigger the S3 source action. However whichever one is detected first starts the CodePipeline, which leads to a race condition in which the image for the new version hasn't been uploaded yet. How can I resolve this? I cannot move S3 out of the Source stage, as per CodePipeline

问题 I'd like to convert a large JSON object to string using the JSON.stringify , but due to the size of the object I got an error of <--- Last few GCs ---> [20817:0x2cc2830] 295727 ms: Scavenge 1335.8 (1423.9) -> 1335.6 (1427.9) MB, 7.7 / 0.0 ms (average mu = 0.255, current mu = 0.170) allocation failure [20817:0x2cc2830] 295966 ms: Mark-sweep 1339.5 (1427.9) -> 1339.3 (1422.9) MB, 227.1 / 0.0 ms (average mu = 0.272, current mu = 0.291) allocation failure scavenge might not succeed [20817

问题 I am reading a dataset dataset1 and dataset2 from S3 locations. I then transform them and write back to the same location where dataset2 was read from. However, I get below error message: An error occurred while calling o118.save. No such file or directory 's3://<myPrefix>/part-00001-a123a120-7d11-581a-b9df-bc53076d57894-c000.snappy.parquet If I try to write to a new S3 location e.g. s3://dataset_new_path.../ then the code works fine. my_df \ .write.mode('overwrite') \ .format('parquet') \

问题 Existing ways of adding content to an S3 file using methods in AmazonS3 class are by putObject with an InputStream Creating a local file with content and uploading it to S3. Is there a way an OutputStream can be created for an existing S3 object to which values from a list can be written into? I see there are no APIs for doing so. 回答1: It's possible to create an S3OutputStream which wraps the AmazonS3 client. See this gist for the implementation: https://gist.github.com/blagerweij

问题 Suppose I have media assets stored in S3 Standard Storage . After 30 days it moved to S3 Standard-Infrequent Access Storage and after 90 days it is moved to Glacier by lifecycle policies that I configured. So at any point of time how can I identify where the object is stored? 回答1: The Storage Class of an object indicates where the data is "stored". For Standard and Standard-Infrequent Access (Standard-IA) , the object is kept as normal in Amazon S3. For Glacier , the object is still listed

问题 Suppose I have media assets stored in S3 Standard Storage . After 30 days it moved to S3 Standard-Infrequent Access Storage and after 90 days it is moved to Glacier by lifecycle policies that I configured. So at any point of time how can I identify where the object is stored? 回答1: The Storage Class of an object indicates where the data is "stored". For Standard and Standard-Infrequent Access (Standard-IA) , the object is kept as normal in Amazon S3. For Glacier , the object is still listed

问题 I need get specific data from gz. how to write the sql? can I just sql as table database?: Select * from gz_File_Name where key = 'keyname' limit 10. but it always turn back with an error. 回答1: You need to create Hive external table over this file location(folder) to be able to query using Hive. Hive will recognize gzip format. Like this: create external table hive_schema.your_table ( col_one string, col_two string ) stored as textfile --specify your file type, or use serde LOCATION 's3:/