When you pass a value by reference the function must dereference it to obtain the value, and each time you modify the value a dereference must occur too since you are writing it in a memory location. I guess compilers are able to understand when something won't be stored back in its reference location so that the value is modified just on registers and stored back just when needed but I'm not sure how much powerful this is.
So there is a indirection step which is not present while passing parameters by value, that can cause worse performance but it's really foggy, since compiler optimizations come into place. Think about the fact that you are passing a pointer, every time you need the value you must fetch the pointer from the stack and then fetch the value pointed (so there are two accesses), while with a normal parameter you have just one.
In any case a reference is used for purposes which are surely different from performance.
