I have a computational geometry problem that I feel should have a relatively simple solution, but I can\'t quite figure it out.
I need to determine the non-convex outlin
Not a fully fleshed-out idea, but anyway: Suppose you started w/ the circular-sweep algorithm for a convex-hull (where you sort, and then process, the points by their angle from a center point). If all of the points end up in this hull, you're done. If not, then you have to "tighten" the hull to include these points. Each of these points were at once candidates for the convex hull, and got removed because they broke the convexness. Sometimes (as with the top purple point in the first example), we can simply leave them in. Where we can't, because the new segment of the hull crosses a segment (as going from the bottom green to the bottom purple in the first example, assuming the bottom aqua point got processed before the green one), the fix is a little more involved (and the part I haven't fleshed out, and is the very part alluded to in the question's latest edit).