Fastest way to sync two Amazon S3 buckets

后端 未结 5 1075
滥情空心
滥情空心 2020-12-15 05:13

I have a S3 bucket with around 4 million files taking some 500GB in total. I need to sync the files to a new bucket (actually changing the name of the bucket would suffice,

5条回答
  •  轻奢々
    轻奢々 (楼主)
    2020-12-15 05:44

    40100 objects 160gb was copied/sync in less than 90 seconds

    follow the below steps:

    step1- select the source folder
    step2- under the properties of the source folder choose advance setting
    step3- enable transfer acceleration and get the endpoint 
    

    AWS configurations one time only (no need to repeat this every time)

    aws configure set default.region us-east-1 #set it to your default region
    aws configure set default.s3.max_concurrent_requests 2000
    aws configure set default.s3.use_accelerate_endpoint true
    

    options :-

    --delete : this option will delete the file in destination if its not present in the source

    AWS command to sync

    aws s3 sync s3://source-test-1992/foldertobesynced/ s3://destination-test-1992/foldertobesynced/ --delete --endpoint-url http://soucre-test-1992.s3-accelerate.amazonaws.com 
    

    transfer acceleration cost

    https://aws.amazon.com/s3/pricing/#S3_Transfer_Acceleration_pricing

    they have not mentioned pricing if buckets are in the same region

提交回复
热议问题