Look at your code again:
temp = ref1;
ref2 = temp;
ref1 = ref2;
References are aliases. All operations that you do on the reference are performed on the object it references. In your code above, how should the compiler magically know that you aren't trying to assign the values to referenced variables, but are rather trying to change the reference itself? There's no syntax to indicate the difference between that, and, for example:
ref1 = 3;
which you'd normally expect to assign a value 3 to the object referenced by ref1.
In short, this is simply by design. If you want "reseatable" references, then you just use pointers (or, if you want to avoid accidential pointer arithmetic, boost::optional).
