I have made a function that finds all the URLs within an html file and repeats the same process for each html content linked to the discovered URLs. The function is recursiv
You could try to wiggle down the nesting by implementing parallel workers (like in cluster computing) instead of increasing the number of nesting function calls.
For example: you define a limited number of slots (eg. 100) and monitor the number of "workers" assigned to each/some of them. If any slots become free, you put the waiting workers "in them".