By REST the URI structure does matter only from human perspective. It does not matter from the perspective of the REST client, because it is following link annotated with semantics.
To answer your question, it matter what do you want to describe, relationships or users. If you want to add and remove relationships, then you have to define a relationship collection. If you want to add and remove users, then you have to define an user collection. This matters only by manipulation. By GET you can define any kind of query which returns a bunch of users...
