Another difference between JVMs is behaviour on undocumented API. (e.g. com.sun.xxx)
For example, Sun's JVM and IBM's JVM both have slightly different behaviour on signal handling. (IBM's JVM doesn't allow the application to trap the "INT" signal in certain cases.)