Copying a 1TB sparse file

Question

I got a sparse file of 1TB which stores actually 32MB data on Linux.

Is it possible to \"efficiently\" make a package to store the sparse file? The package should be

Answer 1

I realize this question is very old, but here's an update that may be helpful to others who find their way here the same way I did.

Thankfully, mvp's excellent answer is now obsolete. According to the GNU tar release notes, SEEK_HOLE/SEEK_DATA was added in v. 1.29, released 2016-05-16. (And with GNU tar v. 1.30 being standard in Debian stable now, it's safe to assume that tar version ≥ 1.29 is available almost everywhere.)

So the way to handle sparse files now is to archive them with whichever tar (GNU or BSD) is installed on your system, and same for extracting.

Additionally, for sparse files that actually contain some data, if it's worthwhile to use compression (ie the data is compressible enough to save substantial disk space, and the disk space savings are worth the likely-substantial time and CPU resources required to compress it):

• tar -cSjf .tar.bz2 /path/to/sparse/file will both take advantage of tar's SEEK_HOLE functionality to quickly & efficiently archive the sparse file, and use bzip2 to compress the actual data.
• tar --use-compress-program=pbzip2 -cSf .tar.bz2 /path/to/sparse/file, as alluded to in marcin's comment, will do the same while also using multiple cores for the compression task.

On my little home server with a quad-core Atom CPU, using pbzip2 vs bzip2 reduced the time by around 25 or 30%.

With or without compression, this will give you an archive that doesn't need any special sparse-file handling, takes up approximately the 'real' size of the original sparse file (or less if compressed), and can be moved around without worrying about inconsistency between different utilities' sparse file capabilities. For example: cp will automatically detect sparse files and do the right thing, rsync will handle sparse files properly if you use the -S flag, and scp has no option for sparse files (it will consume bandwidth copying zeros for all the holes and the resulting copy will be a non-sparse file whose size is the 'apparent' size of the original); but all of them will of course handle a tar archive just fine—whether it contains sparse files or not—without any special flags.

Additional Notes

1. When extracting, tar will automatically detect an archive created with -S so there's no need to specify it.
2. An archive created with pbzip2 is stored in chunks. This results in the archive being marginally bigger than if bzip2 is used, but also means that the extraction can be multithreaded, unlike an archive created with bzip2.
3. pbzip2 and bzip2 will reliably extract each other's archives without error or corruption.