You'll need to do this just like you would locally with python (if you weren't using shutils).
Combine os.walk(), with sftp.mkdir() and sftp.put(). You may also want to check each file and directory with os.path.islink() depending on whether you want to resolve symlinks or not.