For a dataset of 200M GPS (lon, lat) coordinates of vessels I want to calculate an approximate distance to the nearest land or coastline, as a function called distance_to_sh
The key here is that you need to use the "great circle" (orthodromic) distance calculations, which are designed to find the distance between two points on the surface of a sphere. Although the earth is not a perfect sphere, such calculations will get you very close (to within 0.5%), and non-spherical adjustments can be applied if this isn't close enough.
There are many documentations of this formula on the internet. You will want to look for closed form solutions that involve X-Y-Z instead of polar coordinates, or convert your GPS coordinates into polar, one of the two.