I\'m looking for advice on the best way to proceed. I\'m trying to find whether a given point A:(a, b) is inside a regular hexagon, defined with center O:(x, y) and diameter
Subtract the position of the center of the hexagon from your point P to get a vector V. Then, take the dot product of V with the following vectors, which correspond to the three pairs of opposing hexagon edges:
[0,1] ; the edges that are flat with the horizontal
[cos(30),sin(30)] ; the upper-right and lower-left edges
[cos(-30),sin(-30)] ; the lower-right and upper-left edges
If any of the dot products are greater in magnitude than the distance from the center of the hexagon to one of its edges, then the point is not inside the hexagon.
For reference, the dot product of vectors [a,b] and [c,d] is a*c+b*d.
The angle "30" above is in degrees ;)