Sometimes classes are referencing other classes. Implementing std::swap() for such classes cannot be straightforward, because it would lead to swapping of origi
Your program using union is not legal, as you assign the reference but swap the integer than intend to use the reference again. You can read about why, here: Accessing inactive union member and undefined behavior?
An easy solution here is to use pointers instead of references. Then everything will work smoothly, with no special code. If you really don't like this, perhaps you can use boost::optional instead of plain references, but I would struggle to see how that'd be better.