A useful metric for determining when the JVM is about to get into memory/GC trouble

Question

I have a scala data processing application that 95% of the time can handle the data thrown at it in memory. The remaining 5% if left unchecked doesn\'t usually hit OutOf

Answer 1

In addition to the MemoryMXBean notification mechanisms described in @Alla's link. You can use a combination of weak references and reference queues. This old but valid article has a good description of weak, soft, and phantom references and reference queues.

The basic idea is to create a large array (to reserve memory) create a weak or soft reference to it and when doing so, add it to a reference queue. When memory pressure triggers the collection of the weakly referenced array, you will get the reserve memory (breathing life into your application hopefully and giving it time). Have a thread polling the reference queue to determine when your reserve has been collected. You can then trigger the file streaming behavior of your application to finish the job. SoftReferences are more resilient to memory pressure than WeakReferences and make serve your purposes better.