Short answer: use an unassigned user port
Over achiever's answer - Select and deploy a resource discovery solution. Have the server select a private port dynamically. Have the clients use resource discovery.
The risk that that a server will fail because the port it wants to listen on is not available is real; at least it's happened to me. Another service or a client might get there first.
You can almost totally reduce the risk from a client by avoiding the private ports, which are dynamically handed out to clients.
The risk that from another service is minimal if you use a user port. An unassigned port's risk is only that another service happens to be configured (or dyamically) uses that port. But at least that's probably under your control.
The huge doc with all the port assignments, including User Ports, is here: http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt look for the token Unassigned.
