问题
Say I have a website that return me JSON data when I send a GET request using curl. I want to re-direct the output of curl to AWS S3. A new file should be created on S3 for it.
Currently I am able to redirect the output to store it locally.
curl -s -X GET 'http://website_that_returns_json.com' > folder_to_save/$(date +"%Y-%m-%d_%H-%M.json")
I have AWS CLI and s3cmd installed. How would I redirect the output of create to create a new file on AWS S3 ?
Assume :
- AWS S3 access key and secret key are already set.
- Location to store file :
mybucket/$(date +"%Y-%m-%d_%H-%M.json"
回答1:
The AWS Command-Line Interface (CLI) has the ability to stream data to/from Amazon S3:
The following
cpcommand uploads a local file stream from standard input to a specified bucket and key:
aws s3 cp - s3://mybucket/stream.txt
So, you could use:
curl xxx | aws s3 cp - s3://mybucket/object.txt
However, it's probably safer to save the file locally and then copy it to Amazon S3.
回答2:
In case you'd like to run the command on the remote, use aws ssm send-command.
Then to redirect the output of that command to S3, you can use --output-s3-bucket-name parameter.
Here is Bash script to run PowerShell script on the remote and upload it into S3 bucket:
instanceId="i-xyz"
bucketName="bucket_to_save"
bucketDir="folder_to_save"
command="Invoke-WebRequest -UseBasicParsing -Uri http://example.com).Content"
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text --output-s3-bucket-name "$bucketName" --output-s3-key-prefix "$bucketDir" --parameters commands="'${command}'")
while [ "$(aws ssm list-command-invocations --command-id "$cmdId" --query "CommandInvocations[].Status" --output text)" == "InProgress" ]; do sleep 1; done
outputPath=$(aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[].CommandPlugins[].OutputS3KeyPrefix" --output text)
echo "Command output uploaded at: s3://${bucketName}/${outputPath}"
aws s3 ls "s3://${bucketName}/${outputPath}"
To output the uploaded S3 files, run:
aws s3 ls s3://${bucketName}/${outputPath}/stderr.txt && aws s3 cp --quiet s3://${bucketName}/${outputPath}/stderr.txt /dev/stderr
aws s3 cp --quiet s3://${bucketName}/${outputPath}/stdout.txt /dev/stdout
来源：https://stackoverflow.com/questions/42284178/redirect-output-of-console-to-a-file-on-aws-s3