I\'m looking for advice on the best way to proceed. I\'m trying to find whether a given point A:(a, b) is inside a regular hexagon, defined with center O:(x, y) and diameter
You can use the equations for each of the sides of the hexagon; with them you can find out if a given point is in the same half-plane as the center of the hexagon.
For example, the top-right side has the equation:
-sqrt(3)x - y + sqrt(3)/2 = 0
You plug in this the coordinates of the point and then the coordinates of the center. If the results have the same sign, then the point is in the bottom-left half-plane (so it may be inside the hexagon).
You then repeat by using the equations of the others sides.
Note that this algorithm will work for any convex polygon.