Lets say I have something like this (client side code):
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
@Override
The accepted answer is extremely incorrect. It doesn't cryptographically verify any connection between the server certificate and the trusted certificate authority. In general, you should almost never need to implement your own TrustManager, doing so is extremely dangerous.
As EJP stated, there's no need to implement your own TrustManager, you can just use the default one, and ensure that the trusted CA certificate has been added to your default TrustStore. See this question for more information.
Take a look at the CertPathValidator class from the JDK, which verifies a continuous chain of trust from the server's own certificate up through a trusted CA. See Oracle's docs for an introduction to certificate chain validation.