While creating custom exceptions, If we want to create a checked Exception we extend the Exception class and for unchecked exception we extend the RuntimeException class. My question is, how JVM handles subClasses of RuntimeException and Exception differently when they all are sub classes of the Exception class.
It doesn't. The only difference is in requirements enforced by the compiler.
You are mistaken that the JVM handles the exceptions differently, but your question is still valid if you are asking how the compiler treats them differently.
And this has a simple answer: the rule does not state that all subclasses of Exception are checked exceptions. Those which are also subclasses of RuntimeException are unchecked.
来源：https://stackoverflow.com/questions/24798929/how-does-jvm-handles-runtimeexceptions