First of all, System.gc() might as well do nothing. You really can't rely on it to do a garbage collection the way you are suggesting.
Second, you'll want to monitor what is actually going on the with GC by using
-verbosegc -XX:+PrintGCDetails
in your invocation of java. Or by using JConsole, which it sounds like you're doing. But that System.gc() has me scared that you're counting the wrong thing...
I suspect that when you say that the second or third garbage collection is when it frees the memory, you're just miscounting the garbage collections. A request to GC is not a GC! So check the logs (interpret them this way) that PrintGCDetails prints out.
In fact my problem is that I need to run several CITRIX clients sessions on a server, but I would like the running JVMs on the server to free the memory as soon as possible (I have only few high consuming memory functions in my application).
While your problem is valid, the solution you're going for is a little shady. The JVM needs a heap size for exactly this reason - so that it can be guaranteed this space to run in. It seems like you're leaning towards launching an app, then waiting for the JVM to size its heap down, then launching another such that you're overbooking the resources on the machine. Don't do that, because it will all blow up once an app is taking more memory than you figured it would, but which it is entitled to.
I fully believe that you don't want to be micro managing Java's heap this way.
Read enough of http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html to understand the generations and what the trade offs of a larger/smaller heap are.
