Louis Brandy wrote two blog posts analyzing exactly this issue:
More Cache Craziness and Computational Performance - A beginners case study with some interesting statistics and attempts to explain the behavior in more detail, it does indeed come down to cache size limitations.