This is working as intended.
According to your question, you stored the data in this pattern:
POINT (-0.120875610750927 54.1165118880234)
then you claimed that the lat/long is reversed according to the MSDN documentation of
Point(Lat, Long, SRID).
You may realize that the syntax you're using is not the same as the one you claim:
POINT(aValue anotherValue) vs Point(Lat, Long, SRID)
Now, the question is, what does MS SQL do to the data?
Turns out, MS SQL interprets the data as Open Geospatial Consortium (OGC) Well-Known Text (WKT), and thus use STPointFromText function since the format is the most suitable for 2-D point:
POINT(x y)
Now, the follow-up question, does it mean POINT(Lat Long)?
From the sample code
SET @g = geography::STPointFromText('POINT(-122.34900 47.65100)', 4326);
it should be clear that the first parameter is not latitude, but longitude (the range of latitude range is from -90 to 90 only), so now we guess that the format is POINT(Long Lat) then. But why?
As explained in this article,
As you can see [...], the longitude is specified first before the latitude. The reason is because in the Open Geospatial Consortium (OGC) Well-Known Text (WKT) representation, the format is (x, y). Geographic coordinates are usually specified by Lat/Long but between these two, the X is the Longitude while the Y is the Latitude.
You may be wondering why the X-coordinate is the Longitude while the Y-coordinate is the Latitude. Think of the equator of the earth as the x-axis while the prime meridian is the Y-axis. Longitude is defined as the distance from the prime meridian along the x-axis (or the equator). Similarly, latitude is defined as the distance from the equator along the Y-axis.