I have written a recursive function of the form
foo=function(vars,i=2){
**do something with vars**
if(i==length(vars)){
return(**something**)
}else
I guess a site like this is not the right place for a general crash-course on computer science, but in your case there is a match between the site name and the question: You are experiencing a stack overflow! :-)
In computer science, a stack is a data structure where you can access only its last element (a half-queue, so to say). For more information see e.g. Wikipedia or CMU. Stacks play a central role when calling functions, because the return address and often function arguments are stored there. Returning from a function simply means retrieving the return address from the stack and continuing the program execution from the point in code specified by that address.
Since you are applying recursion in your code (calling a function from within itself), the stack grows with every new call. Eventually, your program runs out of memory for storing the whole stack.
See also Memory in R documentation.