I\'m looking for advice on the best way to proceed. I\'m trying to find whether a given point A:(a, b) is inside a regular hexagon, defined with center O:(x, y) and diameter
I would first check if the point is inside the inscribed circle (you can compute the inscribed circle radius easily) or outside the circumscribed circle (that you already have).
The first means the point is in, the latter means it's out.
Statistically, most of the input points should allow you to decide based on the above simple tests.
For the worst case scenario (point is in between the inscribed and circumscribed circles), I think you can find the two vertices that are closest to the point and then see on which side of the segment V1V2 the point is (inner or outer, as relative to the O center). Special case: point is equal to one of the vertices => it's in.
If I'll have a more clever idea (or if I'll ever start to really learn trigonometry), I'll edit the answer to let you know :)