How to copy a large file in Windows XP?

Question

I have a large file in windows XP - its 38GB. (a VM image)

I cannot seem to copy it.

Dragging on the desktop - gives error of "Insufficient system resources exist to complete the requested service"

Using Java - FileChannel.transferTo(0, fileSize, dest) fails for all files > 2GB

Using Java - FileChannel.transferTo() in chunks of 100Mb fails after ~18Gb

java.io.IOException: Insufficient system resources exist to complete the requested service
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:44)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
at sun.nio.ch.IOUtil.write(IOUtil.java:28)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:198)
at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:439)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:510)

I mean - the computer has 3GB of RAM. A 100GB buffer should be enough!?!?

Apparently the DOS commands "copy" and "xcopy" also fail.

(edit) I've tried COPY & XCOPY - these fail with the same error. XCOPY seems to take a really really long time about it too.

I've heard of Robocopy, but it doesn't copy single files?

I'm really feeling that Windows is for the lose right now. Surely microsoft have heard of files larger than a few GB?

Thanks!

Answer 1

In Java, don't try to copy the whole file in a single operation. The transferTo() method works on chunks of a file; wasn't intended as a high-level file copy method. Invoke transferTo() in a loop, and assume that count bytes of data will be in RAM (i.e., lower that parameter to be comfortable fitting in RAM).

FileChannel src = ... 
FileChannel dst = ...
final long CHUNK = 16 * 1024 * 1024; /* 16 Mb */
for (long pos = 0; pos < fileSize; ) {
  pos += src.transferTo(pos, CHUNK, dst);
}

The comment in the transferTo() JavaDoc about it being "more efficient than a simple loop" refers to the fact that channel-to-channel communication can be optimized more than channel-to-user-space-to-channel. It doesn't mean that all looping can be avoided.

Answer 2

I am a Vmware ESX user, I have 30 production VM's with the largest being 232GB. I backup my VM instances onto an internal SATA drive and then copy these off once a week to an external eSata. I use teracopy (free), it runs on average at 45MB/s on an XP machine with 3GB.

Hope that helps Sailen

Answer 3

Well - I've not managed to find a way that works.

None of the packaged tools in windows will copy the file. Drag and drop, COPY, XCOPY, java - all fail to copy the file.

The reason I wanted to copy the file was for a backup before doing an OS upgrade.

In the end i booted into knoppix and copied it.

Answer 4

Take a look at this Hotfix, worth a try as everything I have seen points to this as being a cure for your issue.

EDIT: You can also try XCOPY /Z as pointed out here.

Answer 5

There may be a hardware issue as well.. I suspect you don't have much time, however you may try dumber stream solution and don't set large buffers (8-16MB should be enough):

public static void copy(InputStream input, OutputStream output) throws IOException {
     byte[] buffer = new byte[1024 * 1024 * 8]; // 8MB
     int n = 0;
     while (-1 != (n = input.read(buffer))) {
         output.write(buffer, 0, n);
     }
}

public static void main(String args[]) {

    if (args.length != 2) {
        System.err.println("wrong argument count");
        System.exit(1);
    }

    FileInputStream in = null;
    FileOutputStream out = null;

    try {
        in = new FileInputStream(new File(args[0]));
        out = new FileOutputStream(new File(args[1]));
        copy(in, out);
    } catch (Exception e) {
        e.printStackTrace();
    }

    if (in != null) { try { in.close(); } catch (Exception e) {}}
    if (out != null) { try { out.close(); } catch (Exception e) {}}

}

Answer 6

are you sure the filesystem is actually able to cope with such big files (FAT32 cannot for example)? Take a look on this link for details http://www.ntfs.com/ntfs_vs_fat.htm

The system is 32 or 64 bit? On 32-bit you may have problems copy-ing files larger that 2-4Gb.

Also, you said that rsync scucks for you. I've had a very nice experience with it, copying between 2 hard drives at near-native speed. I've had lots of small files..you seem to have on big blob instead.

You may also try splitting the big blob into smaller blobs:)

Answer 7

final long CHUNK = 16 * 1024 * 1024; /* 16 Mb */
for (long pos = 0; pos < fileSize; pos++) {   
     pos += src.transferTo(pos, CHUNK, dst); 
} 

This does work! just make sure your src and dst are FileChannel objects (input, output respectively)

Answer 8

Another possible answer is Files.copy (java NIO 2), e.g.:

Path sourcePath      = Paths.get("big-file.dat");
Path destinationPath = Paths.get("big-file-copy.dat");

try {
    Files.copy(sourcePath, destinationPath,
            StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
    // something else went wrong
    e.printStackTrace();
}