I have three 3D points like p1(x1,y1,z1), p2(x2,y2,z2), p3(x3,y3,z3).
I have another point, but I know only x, y
p1, p2, p3 define a plane. You can represent it by a point and a normal. For instance, P=p1, N=(p2-P) x (p3-P) (that is, N = cross product of p1p2 and p1p3).
Now for p4 to be in the same plane, it satisfies the plane equation:
(p4-P) · N = 0 %// dot product
⇒ (x4-x1)*N.x + (y4-y1)*N.y + (z4-z1)*N.z = 0
Re-arranging:
z4 = z1 - ((x4-x1)*N.x + (y4-y1)*N.y)/ N.z
No linear system to solve, you just need a cross product.
This is to support both MBo's and Konstantin's answers. Please don't accept this question, but one of the others.
This is how you would implement a solution in MATLAB:
%// Your known 3 points
p1 = [ 1 10 0]';
p2 = [-1 10 10]';
p3 = [ 0 0 10]';
%// your 4th target point
p4 = [0 5 NaN]';
%// Difference matrix/vector
A = [p2-p1 p3-p1];
b = p4-p1;
%// Compute solution
p4(end) = p1(end) + A(3,:)*(A(1:2,:)\b(1:2));
Now, in C++, the mere fact of including the relevant eigen libraries blows up the executable size rather spectacularly. What eigen is capable of is complete overkill for this simple 2x2 system.
So I wouldn't go as far as resort to eigen, unless you have tons of other linear algebra things to do. It is a simple 2x2 system, which is easy enough to solve by hand.
Just KISS it; see DanielKO's answer :)
The mathematical problem here is to solve the following system of equations
p1 + a * (p2 - p1) + b * (p3 - p1) = (x4, y4, z4)
or equivalently
(x1,y1,z1) + a * (x2-x1, y2-y1, z2-z1) + b * (x3-x1, y3-y1, z3-z1) = (x4, y4, z4)
for a, b, and z4.
To solve it in C/C++, you could either implement the Gauss algo (see also the Numerical Recipes book, it is available online), or use Linear Algebra libraries, such as Eigen, or others.
Remark: the approach is the same regardless if the point (x4, y4) lies within the triangle (x1, y1), (x2, y2), (x3, y3), or not.
You can express P4 coordinates in the P1P2P3 vector basis.
x4 = x1 + A * (x2 - x1) + B * (x3 - x1)
y4 = y1 + A * (y2 - y1) + B * (y3 - y1)
This is easy-to-solve linear equation system. You have to find A and B coefficients, then use them to calculate z-coordinate
z4 = z1 + A * (z2 - z1) + B * (z3 - z1)