I am trying to make a map where a user can outline any shape they would like. But I am running into an issue where users can select points that will make the lines of the p
A convex hull might include areas that the user wishes to exclude. Here is another way to approach this that might give more satisfactory results. Check each line to see which ones cross (there are lots of ways to do that). Then reverse the subsequence of points that appear between those two lines.
For example, suppose you are given points A-B-C-D-E-F-A, where B-C and E-F cross. You can uncross them by reversing the subsequence C..E resulting in A-B-E-D-C-F-A.
It's something to try anyway.