I am encountering the following problems trying to work through the quagmire of settings in WCF...
I created a WCF client-server service using a NetTcp binding. I di
The default client credential type for NetTcpBinding is Windows Authentication. For Windows Authentication to work both client and server must be in the same domain, or mutually trusting domains (which in your case you do not have).
If both client and server were on the same domain, WCF would handle the mechanics of Windows Authentication "behind the scenes". And when both client and server are on the same machine they are effectively within the same domain, so Windows can use its own mechanisms to handle the encryption and decryption. It will only do this within mutually trusting domains, though.
If you don't have mutually trusting client and server domains, then the client and server must have some other way to determine if they trust each other with their keys. That's where certificates come in. The client and the server have their own certificates (or the server can issue the client a certificate).
Transport security is like encrypting the outside of the envelope as well as the inside. The downside is if you have to pass the envelope to someone outside your own organization, they need a decryption key just to know where the envelope is supposed to go--now they can read the message in the envelope also. On the other hand, transport security is faster--it requires less security overhead data getting passed along with your envelope.
Message security encrypts your message, but the envelope can be read by the postal workers (the internet and its routers). Only the source and the destination have the keys to decrypt the message, but the intermediaries can properly route your message.
To summarize: to use encryption over the NetTcpBinding both client and server must be within a domain (or mutually trusting domains) or you must have a key exchanging certificate.
Edit: I was asked for some example code--here is a binding element in XAML. It would normally be placed within a netTcpBinding element.
The important part is the security element. For transport security one would change the mode attribute to "Transport". More than likely the clientCredentialType would not be "None" but rather "Certificate", "Ntlm", or "Windows" depending on the context.