Imagine that a server is serving public keys of the users to their partners to make encrypted communication possible. However, the server does NOT have access to the private key
This is the primary problem with public key encryption. You don't have any way to verify that the public key you receive is actually the public key for your intended recipient. The way HTTPS/SSL gets around this is through the use of trusted certificate authorities. The certificate authority signs the public key of the party in question with their private key, guaranteeing that the public key hasn't been changed since it was provided to the certificate authority. Even then, it is only guaranteed that the key provided to you when you request it is the same key that was originally provided to the certificate authority. However, if the server providing those certificates is compromised, you're still in trouble. Even having the server sign the keys as suggested above isn't fool proof if the sever itself is compromised. Ultimately, the security if your key distribution server must be maintained for this system to work.