This is a language limitation as @MikeSeymour pointed out.
And this is the reason why I think std::reference_wrapper should have an overloaded operator& that returns the address of the wrapped object:
template<class T>
struct reference_wrapper{
...
T* operator &(){return ptr;}
}
So that later one can use &Foo->f() or (*&Foo).f() instead of Foo.get().f() or static_cast<T&>(Foo).f().