How can I limit the cache used by copying so there is still memory available for other cache?

Question

Basic situation:

I am copying some NTFS disks in openSuSE. Each one is 2TB. When I do this, the system runs slow.

My guesses:

Answer 1

The nocache command is the general answer to this problem! See https://github.com/Feh/nocache or find it in Debian and Ubuntu 13.10 (saucy).

Thanks, Peter, for alerting us to the --drop-cache" option in rsync. But that was rejected upstream (Bug 9560 – drop-cache option), in favor of a more general solution for this: the new "nocache" command based on the rsync work with fadvise.

You just prepend "nocache" to any command you want. It also has nice utilities for describing and modifying the cache status of files. E.g. here are the effects with and without nocache:

$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]
$ ./nocache cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]\
$ cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 1945/1945 (100.0%)  [filesize=7776.2K, pagesize=4K]

So hopefully that will work for other backup programs (rsnapshot, duplicity, rdiff-backup, amanda, s3sync, s3ql, tar, etc) and other commands that you don't want trashing your cache.