OOP does not originate from any strict formalism, but it is a formalism indeed. There were a number of attempts to define that formalism properly. Most notable work is done by Luca Cardelli: http://lucacardelli.name/indexPapers.html (see the whole "Objects" section)
Imperative programming could be based on any Turing-equivalent formalism, including lambda calculus, SK logic, Turing abstract machine, Markov algorithms, or any other similar Term Rewriting System (TRS). Generic programming is not any different, it is a term rewriting system of a sort.
So, for the most common mathematical grounds for literally everything you'd need to dig into term rewriting systems.
A more recent work is AbdelGawad's recent work at Rice University. He builds a mathematical model of mainstream OOP (eg, Java, C#, C++, Scala, X10, etc) called NOOP. Here is a link to his PhD thesis http://scholarship.rice.edu/handle/1911/70199