How to copy from one bucket to another bucket in s3 of certain suffix

问题

I have 3 buckets 1.commonfolder 2.jsonfolder 3.csvfolder.

• Common folder will be having both json and csv files

• need to copy all csv files to csvfolder

• need to copy all json files to json folder

Code is below to get all the files from commonfolder How to copy after that

import boto3

s3 = boto3.client('s3')
def lambda_handler(event, context):
    #List all the bucket names
    response = s3.list_buckets()
    for bucket in response['Buckets']:
        print (bucket)
        print(f'{bucket["Name"]}')
        #Get the files of particular bucket
        if bucket["Name"] == 'tests3json':
         
            resp = s3.list_objects_v2(Bucket='commonfolder')
            for obj in resp['Contents']:
                files = obj['Key']
                print(files)

            if(filename.split('.')[1].lower()=='json'):
                copyjson(bucket,filename)
                #copyjson(jsonfolder,filename)
            elif(filename.split('.')[1].lower()=='csv'):
                copycsv(bucket, filename)
                #copycsv(csvfolder,filename)

• need to create a new function copyjson,copycsv to do this job

• Need to copy from common-bucket to either csv-bucket or json-bucket depending on the file extension

回答1:

You can check the following code:

import boto3

s3 = boto3.resource('s3')

def lambda_handler(event, context):
    
    source_bucket = s3.Bucket('01-commonfolder-231')
    json_bucket = s3.Bucket('02-jsonfolder-3435')
    csv_bucket = s3.Bucket('03-csvfolder-4552')
    
    for object in source_bucket.objects.all():
        
        #print(object)
        
        if object.key.endswith('.json'):
            
            print(f"{object.key} to json bucket")
      
            copy_object = json_bucket.Object(object.key)
            copy_object.copy({'Bucket': object.bucket_name,
                              'Key': object.key})                             
            
        elif object.key.endswith('.csv'):
            
            print(f"{object.key} to csv bucket")            

            copy_object = csv_bucket.Object(object.key)
            copy_object.copy({'Bucket': object.bucket_name,
                              'Key': object.key})

I tested this using my own sample buckets with test files:

aaa.json to json bucket
bbbbb.csv to csv bucket
bbbbb.json to json bucket
hhhh.csv to csv bucket

回答2:

You can use the move() method from shutil:

from shutil import move
from glob import glob

common_folder = 'C:\\Users\\User\\Desktop\\commonfolder\\'
csv_folder = 'C:\\Users\\User\\Desktop\\csvfolder\\'
json_folder = 'C:\\Users\\User\\Desktop\\jsonfolder\\'

for csv in glob(common_folder+"*.csv"):
    move(csv, csv_folder)

for json in glob(common_folder+"*.json"):
    move(json, json_folder)

来源：https://stackoverflow.com/questions/62587459/how-to-copy-from-one-bucket-to-another-bucket-in-s3-of-certain-suffix