I really need to make std::vector of std::pair with reference (&) inside, but it breaks inside the function when I try to pu
Since C++14 std::make_pair is defined as
template< class T1, class T2 >
std::pair make_pair( T1&& t, T2&& u );
where V1 and V2 are std::decay and std::decay respectively.
This means that your make_pair calls do not really produce pairs with references as their first elements (contrary to what you apparently believed). They actually produce temporaries of pair type. At this point you lose any attachment to the original int object stored in your unique_ptr.
When you pass these pair temporaries to push_back, they get implicitly converted to temporaries of pair type, which is your vector's element type. Through this mechanism you attach references inside your vector's element to int members of those pair temporaries produced by make_pair (not to int objects stored in your unique_ptrs). Once the temporaries expire, the references go sour.
In this case you can eliminate this specific problem by avoiding make_pair altogether and simply directly constructing std::pair objects of proper type, e.g.
vector.push_back(std::pair(*ptr, 11));
but you might run into other problems caused by raw references later.