amazon-s3

问题 I am using: aws s3api list-objects --endpoint-url https://my.end.point/ --bucket my.bucket.name --query 'Contents[].Key' --output text to get the list of files in a bucket. The aws s3api list-object documentation page says that this command returns only up to a 1000 objects, however I noticed that in my case it returns the names of all files in my bucket. For example when I run the following command: aws s3api list-objects --endpoint-url https://my.end.point/ --bucket my.bucket.name --query

问题 Let's say I have the following data structure: / /foo /foo/bar /foo/baz Is it possible to assign to it the following life-cycle rules: / (1 month) /foo (2 months) /foo/bar (3 months) /foo/baz (6 months) The official documentation is unfortunately self-contradictionary in this regard. It doesn't seem to work with AWS console, which makes me somewhat doubtful that SDKs/REST would be any different ;) Failing at that my root problem is: I have 4 types of projects. The most rudimentary type has a

问题 I tried different ways to configure Airflow 1.9 to write logs to s3 however it just ignores it. I found a lot of people having problems reading the Logs after doing so, however my problem is that the Logs remain local. I can read them without problem but they are not in the specified s3 bucket. What I tried was first to write into the airflow.cfg file # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users # must supply an Airflow connection id that provides access to the

问题 I tried different ways to configure Airflow 1.9 to write logs to s3 however it just ignores it. I found a lot of people having problems reading the Logs after doing so, however my problem is that the Logs remain local. I can read them without problem but they are not in the specified s3 bucket. What I tried was first to write into the airflow.cfg file # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users # must supply an Airflow connection id that provides access to the

问题 I tried different ways to configure Airflow 1.9 to write logs to s3 however it just ignores it. I found a lot of people having problems reading the Logs after doing so, however my problem is that the Logs remain local. I can read them without problem but they are not in the specified s3 bucket. What I tried was first to write into the airflow.cfg file # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users # must supply an Airflow connection id that provides access to the

问题 I am using FastImage for caching image and it loads image very fast after caching data as expected. But my server is generating new uri (s3 presigned url) each time for same image. So, FastImage is considering it as new image and tries to download everytime which affects my app performance. My question is, Is there any optimistic way to render images fast as possible without caching it? 回答1: If you have chance to modify the server side application, you can create Authorization headers instead

问题 I'd like to know if it's possible to check if there are certain files in a certain bucket. This is what I've found: Checking if a file is in a S3 bucket using the s3cmd It should fix my problem, but for some reason it keeps returning that the file doesn't exist, while it does. This solution is also a little dated and doesn't use the doesObjectExist method. Summary of all the methods that can be used in the Amazon S3 web service This gives the syntax of how to use this method, but I can't seem

问题 I'm new to using AWS S3. I want to know if it's possible to host multiple static websites in one bucket using the website route directing meta data option. I am planning to have multiple folders each with their own index.html , but how can I configure the bucket settings to route to each individual site when a user types the address. For example by typing http://<bucket-name>.s3-website-<AWS-region>.amazonaws.com/folder1 will take them to website 1 and http://<bucket-name>.s3-website-<AWS

问题 I am running a large Spark 2.1.0 that ends with writing results to s3. It runs on a 30 node cluster and for the most part works fine. However, occasionally I have to stop the job and run it again because a single node gets stuck while writing even after all the computation is done. I am wondering whether I can mitigate this issue by turning speculation on. I read in another post that this may be harmful and lead to duplicate results or data corruption. Can anyone advise ? I was also advised

问题 A user uploads multiple files into my S3 bucket with the current day as prefix for all the files. I need to trigger a lambda function only after I have received all the files under the prefix. How can I do that?. 回答1: Create a DynamoDb table to keep track of the uploaded parts. You should use a HASH key to store the prefix of the files, or something like that. Another attribute could be a count of parts. On each part uploaded, a lambda will be called and it will update the record at the table