Yesterday when I was running the WebLogic Application Server 11g installer, I encountered a OutOfMemory error, so I Googled for the answer:
java -Xms256m -Xmx512
You can use something like VisualVM, http://java.dzone.com/articles/best-kept-secret-jdk-visualvm&default=false&zid=159&browser=16&mid=0&refresh=0, to monitor your memory usage and you will see the max by where it peaks, and it will give you specific info as to which part of memory is actually full, so you can better optimize your environment.
You may find that some part of memory that you don't think about is actually filling up, and by monitoring it you can see what you need to do to get better performance.