AWS S3: How to check if a file exists in a bucket using bash

问题

I'd like to know if it's possible to check if there are certain files in a certain bucket.

This is what I've found:

Checking if a file is in a S3 bucket using the s3cmd

It should fix my problem, but for some reason it keeps returning that the file doesn't exist, while it does. This solution is also a little dated and doesn't use the doesObjectExist method.

Summary of all the methods that can be used in the Amazon S3 web service

This gives the syntax of how to use this method, but I can't seem to make it work.

Do they expect you to make a boolean variable to save the status of the method, or does the function directly give you an output / throw an error?

This is the code I'm currently using in my bash script:

existBool=doesObjectExist(${BucketName}, backup_${DomainName}_${CurrentDate}.zip)

if $existBool ; then
        echo 'No worries, the file exists.'
fi

I tested it using only the name of the file, instead of giving the full path. But since the error I'm getting is a syntax error, I'm probably just using it wrong.

Hopefully someone can help me out and tell me what I'm doing wrong.

!Edit

I ended up looking for another way to do this since using doesObjectExist isn't the fastest or easiest.

回答1:

Last time I saw performance comparisons getObjectMetadata was the fastest way to check if an object exists. Using the AWS cli that would be the head-object method, example:

aws s3api head-object --bucket www.codeengine.com --key index.html

which returns:

{
    "AcceptRanges": "bytes",
    "ContentType": "text/html; charset=utf-8",
    "LastModified": "Sun, 08 Jan 2017 22:49:19 GMT",
    "ContentLength": 38106,
    "ContentEncoding": "gzip",
    "ETag": "\"bda80810592763dcaa8627d44c2bf8bb\"",
    "StorageClass": "REDUCED_REDUNDANCY",
    "CacheControl": "no-cache, no-store",
    "Metadata": {}
}

回答2:

Note that "aws s3 ls" does not quite work, even though the answer was accepted. It searches by prefix, not by a specific object key. I found this out the hard way when someone renamed a file by adding a '1' to the end of the filename, and the existence check would still return True.

(Tried to add this as a comment, but do not have enough rep yet.)

回答3:

Following to @DaveMaple & @MichaelGlenn answers, here is the condition I'm using:

aws s3api head-object --bucket <some_bucket> --key <some_key> || not_exist=true
if [ $not_exist ]; then
  echo "it does not exist"
else
  echo "it exists"
fi

回答4:

One simple way is using aws s3 ls

exists=$(aws s3 ls $path_to_file)
if [ -z "$exists" ]; then
  echo "it does not exist"
else
  echo "it exists"
fi

回答5:

I usually use set -eufo pipefail and the following works better for me because I do not need to worry about unset variables or the entire script exiting.

object_exists=$(aws s3api head-object --bucket $bucket --key $key || true)
if [ -z "$object_exists" ]; then
  echo "it does not exist"
else
  echo "it exists"
fi

来源：https://stackoverflow.com/questions/41871948/aws-s3-how-to-check-if-a-file-exists-in-a-bucket-using-bash

标签

bash

amazon-web-services

amazon-s3