An Apache Tomcat (Atlassian Confluence) instance is started using the following Java options:
JAVA_OPTS=\"-Xms256m -Xmx512m -XX:MaxPermSize=256m -Djava.awt.h
For example, a native library can easily allocate memory outside Java heap.
Direct ByteBuffer also does that: http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html
The contents of direct buffers may reside outside of the normal garbage-collected heap, and so their impact upon the memory footprint of an application might not be obvious.
There are good reasons to allocate huge direct ByteBuffers.
http://ehcache.org/documentation/offheap_store.html