Why does the speed of memcpy() drop dramatically every 4KB?

Question

I tested the speed of memcpy() noticing the speed drops dramatically at i*4KB. The result is as follow: the Y-axis is the speed(MB/second) and the X-axis is the

Answer 1

I expect it's because:

• When the block size is a 4KB multiple, then malloc allocates new pages from the O/S.
• When the block size is not a 4KB multiple, then malloc allocates a range from its (already allocated) heap.
• When the pages are allocated from the O/S then they are 'cold': touching them for the first time is very expensive.

My guess is that, if you do a single memcpy before the first gettimeofday then that will 'warm' the allocated memory and you won't see this problem. Instead of doing an initial memcpy, even writing one byte into each allocated 4KB page might be enough to pre-warm the page.

Usually when I want a performance test like yours I code it as:

// Run in once to pre-warm the cache
runTest();
// Repeat 
startTimer();
for (int i = count; i; --i)
  runTest();
stopTimer();

// use a larger count if the duration is less than a few seconds
// repeat test 3 times to ensure that results are consistent