Paramiko Fails to download large files >1GB

Question

def download():
if os.path.exists( dst_dir_path ) == False:
    logger.error( \"Cannot access destination folder %s. Please check path and permissions. \" % ( dst_di         


        

Answer 1

I've run into problems downloading large files (>1 GB) via SFTP using pysftp. Underlying library is Paramiko. Googling about the problem lead me here and there are great solutions. Nevertheless, many posts are relatively old and I suppose a majority of these problems have been solved over time. And it did not help with my problem.

Which is: Paramiko runs into a memory error while loading chunks during prefetch in sftp_file.py. The list grows beyond limits and memory error was somehow not blocking execution. It was probably silently consumed some way on the stack. The download fails only when this error happens, and they run in separate threads.

Anyway, the way to control the size of the list is to set the MAX_REQUEST_SIZE:

paramiko.sftp_file.SFTPFile.MAX_REQUEST_SIZE = pow(2, 22) # 4MB per chunk

If you go over 16MBs though, you'll run into a new problem: paramiko.sftp.SFTPError: Garbage packet received. Turns out there is a check in sftp.py in _read_packet method:

# most sftp servers won't accept packets larger than about 32k, so
# anything with the high byte set (> 16MB) is just garbage.
if byte_ord(x[0]):
    raise SFTPError("Garbage packet received")

So if a chunk is > 16MB we have this error raised. I did not care to fiddle with Paramiko library itself, so I had to keep my chunk size at an 'acceptable maximum' at 4MB.

This way I was able to download files of size > 30GB. Hope this helps people.