Moving big file from S3 to SFTP + Check if SFTP file path is directory or file path

问题

Requirement: Move big file from S3 to SFTP

Issue: For the file size of 500MB its taking very long time to upload into SFTP (Able to solve this , please check below EDIT 1:Solution)

Code:

with sftp_client.open(self.sftp_path + key_name, 'wb') as f:
    s3_client.download_file(self.s3_bucket, self.s3_key, f)

I have read the link Reading file opened with Python Paramiko SFTPClient.open method is slow

And I have tried

with sftp_client.open(self.sftp_path + key_name, 'wb') as f:
    s3_client.download_fileobj(self.s3_bucket, self.s3_key, f.prefetch())

Got an error as: Fileobj must implement write

Also tried with buffer, still, it's not fast

with sftp_client.open(self.sftp_path + key_name, 'wb',32768) as f:
    s3_client.download_fileobj(self.s3_bucket, self.s3_key, f)

> EDIT 1: Solution

I was able to upload file faster by using the below approach

Code:

    def execute(self, context):
        self.s3_key = self.get_s3_key(self.s3_key)
        s3_hook = S3Hook(self.s3_conn_id)
        s3_client = s3_hook.get_conn()
        with tempfile.NamedTemporaryFile('w') as f:
            if not s3_hook.check_for_key(self.s3_key, bucket_name=self.s3_bucket):
                logging.info(f'The source key {self.s3_key} does not exist')
            else:
                s3_client.download_file(self.s3_bucket, self.s3_key, f.name)
                sftp_client.put(f.name, self.sftp_path)

But got another issue

For some SFTP, it works fine with the path as sftp_path = /path/win/
For some SFTP, it fails with the path as sftp_path = /path/win/

Error OSError: Specified file is a directory..

And it works fine only with sftp_path = /path/win/file.txt

How can I make sure it works fine in both ways?

来源：https://stackoverflow.com/questions/62903247/moving-big-file-from-s3-to-sftp-check-if-sftp-file-path-is-directory-or-file-p

标签

python-3.x

amazon-s3

sftp