Dereferencing can be expensive mostly because it costs an instruction to fetch data from memory which might be far away and do not exhibit locality of reference. In that case, the processor should fetch data from non-cached memory and even hard disk (in case of a hard page fault).