I have scene composed of one arbitrary quadrilateral. I need to be able to transform that quadrilateral into a rect. Each quad is in 2d coordinates, so they have 4 vertex
Not all quadrilaterals are rectangles. There is no invertible transformation from a quad to a rectangle for this reason; there exist more quads than rects, so you cannot produce an invertible mapping from quads to rects.
However, you can generate an invertible transformation for a particular quadrilateral. As you surmise, it's about rotating the perspective so the quadrilateral "appears" as a rectangle in your new coordinate space. See http://alumni.media.mit.edu/~cwren/interpolator/ , which contains Matlab source code for this problem.