In addition to Earwicker's response which I generally agree with. I would also speculate that since & is the "address-of" operator, it is somewhat fitting. Since a reference in many ways is like passing by address instead of by value.
In addition to that, taking the address of a variable is often referred to as "referencing"
(Yes I know that references don't have to be implemented using pointers under the hood, I am referring to the way they conceptually work).
This is just speculation though.
