Basic situation:
I am copying some NTFS disks in openSuSE. Each one is 2TB. When I do this, the system runs slow.
My guesses:
The kernel can not know, that you won't use the cached data from copying agains. This is your information advantage.
But you could set the swapiness to 0: sudo sysctl vm.swappiness=0. This will cause linux to drop the cache before libraries etc. are written to the swap.
Works nice for me too, especially very performant in combination with hugh ram (16-32 GB).