My Java application runs another Java application, by running the process \"java -jar j.jar\". J.jar is known to use a LOT of memory depending on the dataset it is given, an
Depending on your OS, this might work for getting the free and available memory size:
java.lang.management.OperatingSystemMXBean mxbean = java.lang.management.ManagementFactory.getOperatingSystemMXBean();
com.sun.management.OperatingSystemMXBean sunmxbean = (com.sun.management.OperatingSystemMXBean) mxbean;
long freeMemory = sunmxbean.getFreePhysicalMemorySize();
long availableMemory = sunmxbean.getTotalPhysicalMemorySize();
From there, you can figure out 80-90% and launch your jar with the max memory size you want.
I don't know that this works with all OS's (i.e. Windows), but it worked when I tested it with both OSX and Linux.