Apache Commons Net FTP is uploading corrupted files

Question

I\'m trying to use Apache Commons Net for FTP file transfers.

Problem is files are intermittently arriving at the server corrupt. By \'corrupt\' I mean that WinRAR t

Answer 1

Commons FTP defaults to Ascii file types. You want to set it to Binary when dealing with binary data like a ZIP file.

From http://commons.apache.org/net/api/org/apache/commons/net/ftp/FTPClient.html

The default settings for FTPClient are for it to use FTP.ASCII_FILE_TYPE , FTP.NON_PRINT_TEXT_FORMAT , FTP.STREAM_TRANSFER_MODE , and FTP.FILE_STRUCTURE . The only file types directly supported are FTP.ASCII_FILE_TYPE and FTP.BINARY_FILE_TYPE .

You want to do setFileType(FTP.BINARY_FILE_TYPE) before you send the file.

Answer 2

I had this problem despite specifying binary file type so I wrote code to validate the uploaded file via MD5 hashing:

public void upload(String sourceFilePath) throws Exception
{
    while (true)
    {
        // Upload
        File sourceFile = new File(sourceFilePath);
        String sourceFileHash = MD5Checksum.getMD5Checksum(sourceFilePath);
        String remoteFile = sourceFile.getName();

        try (InputStream inputStream = new FileInputStream(sourceFile))
        {
            boolean successful = ftpClient.storeFile(remoteFile, inputStream);

            if (!successful)
            {
                throw new IllegalStateException("Upload of " + sourceFilePath + " failed!");
            }
        }

        // Download
        File temporaryFile = File.createTempFile("prefix", "suffix");
        try (OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(temporaryFile)))
        {
            boolean successful = ftpClient.retrieveFile(remoteFile, outputStream);

            if (!successful)
            {
                throw new IllegalStateException("Download of " + sourceFilePath + " failed!");
            }
        }

        String downloadFileHash = MD5Checksum.getMD5Checksum(temporaryFile.getAbsolutePath());
        Files.delete(temporaryFile.toPath());

        // Make sure the file hashes match
        if (sourceFileHash.equals(downloadFileHash))
        {
            break;
        }
    }
}

MD5Checksum.java:

import java.io.*;
import java.security.MessageDigest;

public class MD5Checksum
{
    private static byte[] createChecksum(String filename) throws Exception
    {
        try (InputStream fileInputStream = new FileInputStream(filename))
        {
            byte[] buffer = new byte[1024];
            MessageDigest complete = MessageDigest.getInstance("MD5");
            int numRead;

            do
            {
                numRead = fileInputStream.read(buffer);
                if (numRead > 0)
                {
                    complete.update(buffer, 0, numRead);
                }
            } while (numRead != -1);

            return complete.digest();
        }
    }

    public static String getMD5Checksum(String filename) throws Exception
    {
        byte[] checksum = createChecksum(filename);
        StringBuilder result = new StringBuilder();

        for (byte singleByte : checksum)
        {
            result.append(Integer.toString((singleByte & 0xff) + 0x100, 16).substring(1));
        }

        return result.toString();
    }
}

The MD5 code is taken from here.

Answer 3

Solution

I had the same issue and solved it by calling

ftpClient.setFileType(FTP.BINARY_FILE_TYPE)

before each method retrieveFile, retrieveFileStream, storeFile

Explanation

File is corrupted, because default fileType is FTP.ASCII_FILE_TYPE. This causes the issue. If you are on linux all bytes \n\r (windows end of file) are changed into \n byte. And this corrupt the file.

To avoid this behavior you have to call ftpClient.setFileType(FTP.BINARY_FILE_TYPE). Unfortunately, this setup is reset by each connect method back to ASCII_FILE_TYPE. In my case this was reset even by method listFiles. I guess, that this happened because I use passiveMode on ftpClient.

So if you want to avoid troubles call setFileType(FTP.BINARY_FILE_TYPE) right before every file transfer.