I have a S3 bucket with around 4 million files taking some 500GB in total. I need to sync the files to a new bucket (actually changing the name of the bucket would suffice,
40100 objects 160gb was copied/sync in less than 90 seconds
follow the below steps:
step1- select the source folder
step2- under the properties of the source folder choose advance setting
step3- enable transfer acceleration and get the endpoint
AWS configurations one time only (no need to repeat this every time)
aws configure set default.region us-east-1 #set it to your default region
aws configure set default.s3.max_concurrent_requests 2000
aws configure set default.s3.use_accelerate_endpoint true
options :-
--delete : this option will delete the file in destination if its not present in the source
AWS command to sync
aws s3 sync s3://source-test-1992/foldertobesynced/ s3://destination-test-1992/foldertobesynced/ --delete --endpoint-url http://soucre-test-1992.s3-accelerate.amazonaws.com
transfer acceleration cost
https://aws.amazon.com/s3/pricing/#S3_Transfer_Acceleration_pricing
they have not mentioned pricing if buckets are in the same region