Because references don't have an address of their own (if they do that is implementation dependent, and you can't access them with the & operator directly).
The & operator is used to get the address of the thing that the reference is referring to.
Whereas a pointer does have an address of its own and you can access it with the & operator.
int x = 4;
int *p = &x;
//p holds the address of x
//&p holds the address of p.
int &r = x;
//&r holds the address of x
And if something doesn't have an address of its own then it can't store a value inside of it. And if it can't store a value inside of it, then it can't be changed.
I think the parashift C++ FAQ on references says it best:
Important note: Even though a
reference is often implemented using
an address in the underlying assembly
language, please do not think of a
reference as a funny looking pointer
to an object. A reference is the
object. It is not a pointer to the
object, nor a copy of the object. It
is the object.
and again in FAQ 8.5 :
Unlike a pointer, once a reference is
bound to an object, it can not be
"reseated" to another object. The
reference itself isn't an object (it
has no identity; taking the address of
a reference gives you the address of
the referent; remember: the reference
is its referent).