Is this expected with primitive types?
I'd say absolutely. They don't have constructors, so that doesn't need to be called.
Is there no point passing them by reference?
There is: when you want to have output parameters, then, in C++, passing by reference is considered better pracitce than passing a pointer.
I was expecting a copy of a primitive type is made when not using by reference and therefore expected a small performance-boost.
Well, since passing by reference is usually implemented using pointers, then the compiler has to emit code that pushes something onto the stack, either the value, or a pointer to the value - and it really does not matter which one is done.