For the purposes of estimating the maximum call depth a recursive method may achieve with a given amount of memory, what is the (approximate) formula for calculating the mem
Only a partial answer: from JVM Spec 7, 2.5.2, stack frames can be allocated on the heap, and the stack size may be dynamic. I couldn't say for certain, but it seems it should be possible to have your stack size bounded only by your heap size:
Because the Java virtual machine stack is never manipulated directly except to push and pop frames, frames may be heap allocated.
and
This specification permits Java virtual machine stacks either to be of a fixed size or to dynamically expand and contract as required by the computation. If the Java virtual machine stacks are of a fixed size, the size of each Java virtual machine stack may be chosen independently when that stack is created.
A Java virtual machine implementation may provide the programmer or the user control over the initial size of Java virtual machine stacks, as well as, in the case of dynamically expanding or contracting Java virtual machine stacks, control over the maximum and minimum sizes.
So it'll be up to the JVM implementation.