If I create a buffer via ByteBuffer.allocateDirect(), the memory exists outside of the Java heap. Is there a way to measure this kind of memory usage from my a
You can use reflections to get Bits.reservedMemory on OpenJDK/HotSpot Java 7. There is no platform independent way and this only shows you the usage via ByteBuffer.allocateDirect() not any other way of allocating native memory.
An alternative is to parse /proc/{pid}/maps and exclude file mappings. This is an approximation of the virtual memory your process is using.