Remote linux server to remote linux server large sparse files copy - How To?

Question

I have two twins CentOS 5.4 servers with VMware Server installed on each.

What is the most reliable and fast method for copying virtual machines files from one server to the other, assuming that I always use sparse file for my vmware virtual machines?

The vm's files are a pain to copy since they are very large (50 GB) but since they are sparse files I think something can be done to improve the speed of the copy.

Answer 1

If you want to copy large data quickly, rsync over SSH is not for you. As running an rsync daemon for quick one-shot copying is also overkill, yer olde tar and nc do the trick as follows.

Create the process that will serve the files over network:

tar cSf - /path/to/files | nc -l 5000

Note that it may take tar a long time to examine sparse files, so it's normal to see no progress for a while.

And receive the files with the following at the other end:

nc hostname_or_ip 5000 | tar xSf -

---

Alternatively, if you want to get all fancy, use pv to display progress:

tar cSf - /path/to/files \
   | pv -s `du -sb /path/to/files  | awk '{ print $1 }'` \
   | nc -l 5000

Wait a little until you see that pv reports that some bytes have passed by, then start the receiver at the other end:

nc hostname_or_ip 5000 | pv -btr | tar xSf -

Answer 2

Have you tried rsync with the option --sparse(possibly over ssh)?

From man rsync:

    Try  to  handle  sparse  files  efficiently so they take up less
    space on the destination.  Conflicts with --inplace because it’s
    not possible to overwrite data in a sparse fashion.

Answer 3

You could have a look at http://www.barricane.com/virtsync

(Disclaimer: I am the author.)

Answer 4

Since rsync is terribly slow at copying sparse file, I usually resort using tar over ssh :

tar Scjf - my-src-files | ssh sylvain@my.dest.host tar Sxjf - -C /the/target/directory