According to the standard, in [expr.sizeof] (5.3.3.2) we get:
When applied to a reference or a reference type, the result is the size of the reference
The choice is somewhat arbitrary, and trying to fully justify either option will lead to circular metaphysical arguments.
The intent of a reference is to be (an alias for) the object itself; under that reasoning it makes sense for them both to have the same size (and address), and that is what the language specifies.
The abstraction is leaky - sometimes a reference has its own storage, separate from the object - leading to anomolies like those you point out. But we have pointers for when we need to deal with a "reference" as a separate entity to the object.