Sometimes classes are referencing other classes. Implementing std::swap() for such classes cannot be straightforward, because it would lead to swapping of origi
The union trick you're using is about as non-portable as code gets. The standard places no requirements whatsoever on how compilers implement references. They can (and most probably do) use pointers under the hood, but this is never guaranteed. Not to mention the fact that sizeof(size_t) and sizeof(T*) aren't required to be equal anyway.
The best answer to your problem is: don't use reference members if you need an assignable/swappable class. Just use a pointer member instead. After all, references are non-reseatable by definition, yet by wanting the class swappable, you want something reseatable. And that's a pointer.