I want to improve a collision system.
Right now I detect if 2 irregular objects collide if their bounding rectangles collide.
I want to obtain the for recta
Forget about a mathematical solution. As you can easily see by drawing, you can have up to four solutions, and thus likely a fourth grade polynomial.
Instead just do a binary search along the edge of one of the figures. It is easy to determine if a point lies within an ellipse and even more so in a circle (just see if distance is shorter than radius).
If you really want to go for the maths, Wolfram MathWorld has a nice article here: http://mathworld.wolfram.com/Circle-EllipseIntersection.html but be warned, you'll still have to write a polynomial equation solver, probably using something like binary search.