Suppose I have a Swings Java Application, I set the min Heap is 64MB and max Heap 2GB, when the user start the application, the log in screen will be displayed, at this time
I have posted my test results over there. Basically, MaxHeapFreeRatio is not respected by every GC implementation, and, to make it worse, it seems to be necessary that there is enough heap activity in order to trigger it in a timely manner, ie. could be that you require 2 full GC runs to actually release the memory to the OS. And if you have a burst memory footprint of X GB, then you probably have to allocate 1 or 2 times of that amount in order to trigger the heap downsize. Or you call System.gc() manually.
If performance is not an issue, and memory footprint is all that counts, try:
-XX:UseSerialGC -Xms16M -Xminf=5 -Xmaxf=10
I want my application release the memory to the OS when it doesn't need to use a big memory. Can I do it at runtime with java app?
No you can't.
Under some circumstances, the GC will release memory back to the OS of its own accord, but I'm not aware of any JVM that allows an application to tell the GC to do this. And on top of that, the GC is rather conservative about doing this because ... as a general rule ... the JVM will operate more efficiently with more memory, and continually requesting / giving back memory to the OS is inefficient.
Note that the GC tuning option -XX:MaxHeapFreeRatio
can be used to specify the maximum ratio of free to used heap before the GC will give memory back. However, there are complications. For example, not all available GCs respect this option. If you are going to try this approach, I suggest you do some research ... and don't expect miracles.
Try forking parts of your code to a separate thread and dispose it when you don't need it anymore.
The serialgc in hotspot will give memory back... at least it used to. Your performance will nose dive.
All of the garbage collectors in IBM J9 jvm will release memory. I'm not sure this jvm is a free download...
The best answer is, why are you concerned? Memory is dirt cheap these days and free memory is wasted memory. Is this actually a problem? The OS will page the extra unused memory to disk anyway. The best answer is to ignore it. :)
You might also consider Googling off-heap storage, perhaps Teracotta/eh cache.
EDIT:
I just noticed in JDK1.7u6, that using G1GC with a small Xms and large Xmx, the garbage collector will reduce the size of the heap after a few garbage collections where excess objects have been freed.
You can dispose of objects and suggest the garbage collector do its work, but if the GC does not feel it is necessary, the request will be ignored. So in summary, 'no'.
Note that memory assigned to Java apps. is set before the app. starts, and is not adjustable.