I want to improve a collision system.
Right now I detect if 2 irregular objects collide if their bounding rectangles collide.
I want to obtain the for recta
Short answer: Solving exactly for whether the two objects intersect is complicated enough to be infeasible for the purpose of collision detection. Discretize your ellipse as an n-sided polygon for some n (depending on how accurate you need to be) and do collision detection with that polygon.
Long answer: If you insist on determining if the smooth ellipse and circle intersect, there are two main approaches. Both involve solving first for the closest point to the circle's center on the ellipse, and then comparing that distance to the circle's radius.
Approach 1: Use a parametrization of the ellipse. Transform your coordinates so that the ellipse is at the origin, with its axes aligned to the x-y axes. That is:
The equation of the ellipse is then given by a cos(t), b sin(t). To find the closest point, we want to minimize the square distance
|| (a cos t, b sin t) - c ||^2. As Jean points out, this is "just calculus": take a derivative, and set it equal to 0. Unless I'm missing something, though, solving the resulting (quite nasty) equation for t is not possible analytically, and must be approximated using e.g. Newton's Method.
Plug in the t you find into the parametric equation to get the closest point.
t.Approach 2: Use multidimensional calculus. No change of coordinates is necessary.
Finding the point on the ellipse closest to the center of the circle can then be phrased as a constrained minimization problem:
Minimize ||(x,y) - c||^2 subject to g(x,y) = 0
(Minimizing the square distance is equivalent to minimizing the distance, and much more pleasant to deal with since it's a quadratic polynomial in x,y.)
To solve the constrained minimization problem, we introduce Lagrange multiplier lambda, and solve the system of equations
2 * [ (x,y) -c ] + lambda * Jg(x,y) = 0
g(x,y) = 0
Here Jg is the gradient of g. This is a system of three (nonlinear) equations in three unknowns: x, y, and lambda. We can solve this system using Newton's Method, and the (x,y) we get is the closest point to the circle's center.
Caveat: both of these approaches technically solve for the point which extremizes the distance to the circle's center. Thus the point found might be the furthest point from the circle, and not the closest. For both methods, seeding your solve with a good initial guess (the center of the circle works well for Method 2; you're on your own for Method 1) will reduce this danger.
Potential Third Approach?: It may be possible to directly solve for the roots of the system of two quadratic equations in two variables representing the circle and ellipse. If a real root exists, the objects intersect. The most direct way of solving this system, again using a numerical algorithm like Newton's Method, won't help because lack of convergence does not necessary imply nonexistence of a real root. For two quadratic equations in two variables, however, there may exist a specialized method that's guaranteed to find real roots, if they exist. I myself can't think of a way of doing this, but you may want to research it yourself (or see if someone on stackoverflow can elaborate.)