We ship a Java application whose memory demand can vary quite a lot depending on the size of the data it is processing. If you don\'t set the max VM (virtual memory) size
if you have a lot of time on your hand you could try the following :
Try to obtain what is the needed memory vs input dataset. With this you can split processing in a different set of classes and create a new JVM process to actually process the data. Basically a Manager and a Worker. The Manager would do a basic analysis on the demanded dataset and spawn a Worker with the appropriate memory requirements. You could probably also set your Manager to be aware of the environment and warn the user when they are trying to operate on a dataset their machine cannot handle.
This is pretty much an extension on the answer provided by skaffman but will happen all within the same app as far as the user is concerned.