How to display only files from aws s3 ls command?

问题

I am using the aws cli to list the files in an s3 bucket using the following command (documentation):

aws s3 ls s3://mybucket --recursive --human-readable --summarize

This command gives me the following output:

2013-09-02 21:37:53   10 Bytes a.txt
2013-09-02 21:37:53  2.9 MiB foo.zip
2013-09-02 21:32:57   23 Bytes foo/bar/.baz/a
2013-09-02 21:32:58   41 Bytes foo/bar/.baz/b
2013-09-02 21:32:57  281 Bytes foo/bar/.baz/c
2013-09-02 21:32:57   73 Bytes foo/bar/.baz/d
2013-09-02 21:32:57  452 Bytes foo/bar/.baz/e
2013-09-02 21:32:57  896 Bytes foo/bar/.baz/hooks/bar
2013-09-02 21:32:57  189 Bytes foo/bar/.baz/hooks/foo
2013-09-02 21:32:57  398 Bytes z.txt

Total Objects: 10
   Total Size: 2.9 MiB

However, this is my desired output:

a.txt
foo.zip
foo/bar/.baz/a
foo/bar/.baz/b
foo/bar/.baz/c
foo/bar/.baz/d
foo/bar/.baz/e
foo/bar/.baz/hooks/bar
foo/bar/.baz/hooks/foo
z.txt

How can I omit the date, time and file size in order to show only the file list?

回答1:

You can't do this with just the aws command, but you can easily pipe it to another command to strip out the portion you don't want. You also need to remove the --human-readable flag to get output easier to work with, and the --summarize flag to remove the summary data at the end.

Try this:

aws s3 ls s3://mybucket --recursive | awk '{print $4}'

Edit: to take spaces in filenames into account:

aws s3 ls s3://mybucket --recursive | awk '{$1=$2=$3=""; print $0}' | sed 's/^[ \t]*//'

回答2:

A simple filter would be:

aws s3 ls s3://mybucket --recursive | perl -pe 's/^(?:\S+\s+){3}//'

This will remove the date, time and size. Left only the full path of the file. It also works without the recursive and it should also works with filename containing spaces.

回答3:

Use the s3api with jq (AWS docu aws s3api list-objects):

This mode is always recursive.

$ aws s3api list-objects --bucket "bucket" | jq -r '.Contents[].Key'
a.txt
foo.zip
foo/bar/.baz/a
[...]

You can filter sub directories by adding a prefix (here foo directory). The prefix must not start with an /.

$ aws s3api list-objects --bucket "bucket" --prefix "foo/" | jq -r '.Contents[].Key'
foo/bar/.baz/a
foo/bar/.baz/b
foo/bar/.baz/c
[...]

jq Options:

-r = Raw Mode, no quotes in output
.Contents[] = Get Contents Object Array Content
.Key = Get every Key Field (does not produce a valid JSON Array, but we are in raw mode, so we don't care)

Addendum:

You can use pure AWS CLI, but the values will be seperated by \x09 = Horizontal Tab (AWS: Controlling Command Output from the AWS CLI - Text Output Format)

$ aws s3api list-objects --bucket "bucket" --prefix "foo/" --query "Contents[].Key" --output text
foo/bar/.baz/a   foo/bar/.baz/b   foo/bar/.baz/c   [...]

AWS CLI Options:

--query "Contents[].Key" = Query Contents Object Array and get every Key inside
--output text = Output as Tab delimited Text with now Quotes

回答4:

Simple Way

aws s3 ls s3://mybucket --recursive --human-readable --summarize|cut -c 29-

回答5:

My Solution

List only files recursively using aws cli.

aws s3 ls s3://myBucket --recursive | awk 'NF>1{print $4}' | grep .

grep . - Clear empty lines.

Example: aws s3 ls s3://myBucket

                           PRE f5c10c1678e8484482964b8fdcfe43ad/
                           PRE f65b94ad31734135a61a7fb932f7054d/
                           PRE f79b12a226b542dbb373c502bf125ffb/
                           PRE logos/
                           PRE test/
                           PRE userpics/
2019-05-14 10:56:28       7754 stage.js

Solution: aws s3 ls s3://myBucket --recursive | awk 'NF>1{print $4}' | grep .

stage.js

回答6:

Simple command would be

aws s3 ls s3://mybucket --recursive --human-readable --summarize |cut -d ' ' -f 8

If you need the timestamp, just update command field values.

回答7:

For only the file names, I find the easiest to be:

aws s3 ls s3://path/to/bucket/ | cut -d " " -f 4

This will cut the returned output at the spaces (cut -d " ") and return the fourth column (-f 4), which is the list of file names.

回答8:

An S3 bucket may not only have files but also files with prefixes. In case you use --recursive it will not only list the files but also just the prefixes. In case you do not care about the prefixes and just the files within the bucket or the within prefixes within the bucket, this should work.

aws s3 ls s3://$S3_BUCKET/$S3_OPTIONAL_PREFIX/ --recursive | awk '{ if($3 >0) print $4}'

awk's $3 is the size of the file in case of prefix it would be 0. It could also be that the file is empty so it would skip empty files as well.

来源：https://stackoverflow.com/questions/36813327/how-to-display-only-files-from-aws-s3-ls-command

标签

Linux

amazon-web-services

amazon-s3

aws-cli