My desktop is win7 and I am trying to connect to a server 2012.
Both hosts are on the same domain.
If I do:
Enter-PSSession -ComputerName *Serve
After talking to our domain admin I think I found the cause of the problem.
The server is in a resource domain which only has a one way trust to the main OU.
This explains why I can do PS remoting from the server to the clients but not vice versa.
I found that I can use the IP address with the -Credential option in my case though which isn't nice but an acceptable workaround.