Redirect output of console to a file on AWS S3

Question

Say I have a website that return me JSON data when I send a GET request using curl. I want to re-direct the output of curl to AWS S3. A new file should be created on S3 for it.

Currently I am able to redirect the output to store it locally.

curl -s -X GET 'http://website_that_returns_json.com' > folder_to_save/$(date +"%Y-%m-%d_%H-%M.json")

I have AWS CLI and s3cmd installed. How would I redirect the output of create to create a new file on AWS S3 ?

Assume :

1. AWS S3 access key and secret key are already set.
2. Location to store file : mybucket/$(date +"%Y-%m-%d_%H-%M.json"

Answer 1

The AWS Command-Line Interface (CLI) has the ability to stream data to/from Amazon S3:

The following cp command uploads a local file stream from standard input to a specified bucket and key:

aws s3 cp - s3://mybucket/stream.txt

So, you could use:

curl xxx | aws s3 cp - s3://mybucket/object.txt

However, it's probably safer to save the file locally and then copy it to Amazon S3.

Answer 2

In case you'd like to run the command on the remote, use aws ssm send-command.

Then to redirect the output of that command to S3, you can use --output-s3-bucket-name parameter.

Here is Bash script to run PowerShell script on the remote and upload it into S3 bucket:

instanceId="i-xyz"
bucketName="bucket_to_save"
bucketDir="folder_to_save"
command="Invoke-WebRequest -UseBasicParsing -Uri http://example.com).Content"
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text  --output-s3-bucket-name "$bucketName" --output-s3-key-prefix "$bucketDir" --parameters commands="'${command}'")
while [ "$(aws ssm list-command-invocations --command-id "$cmdId" --query "CommandInvocations[].Status" --output text)" == "InProgress" ]; do sleep 1; done
outputPath=$(aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[].CommandPlugins[].OutputS3KeyPrefix" --output text)
echo "Command output uploaded at: s3://${bucketName}/${outputPath}"
aws s3 ls "s3://${bucketName}/${outputPath}"

To output the uploaded S3 files, run:

aws s3 ls s3://${bucketName}/${outputPath}/stderr.txt && aws s3 cp --quiet s3://${bucketName}/${outputPath}/stderr.txt /dev/stderr
aws s3 cp --quiet s3://${bucketName}/${outputPath}/stdout.txt /dev/stdout