Why does this MySQL stored function give different results than to doing the calculation in the query?

Question

This is a question about calculating the distance between two points of latitude and longitude on the earth using a haversine formula, for use in projects where you need to have a 'find my nearest' function.

The haversine formula is well discussed and solved in MySQL in this post.

I then asked this question about turning it into a stored function so that its available for future projects without having to lookup, remember or re-type the formula in its long form.

Its all good. Except my function differs in results (slightly) to just typing the formula directly into the query, all other things being equal. Why is this?

So here's the function I wrote:

DELIMITER $$

DROP FUNCTION IF EXISTS haversine $$

CREATE FUNCTION `haversine`
    (fromLatitude FLOAT,
     fromLongitude FLOAT,
     toLatitude FLOAT,
     toLongitude FLOAT,
     unit VARCHAR(20)
     )
    RETURNS FLOAT
    DETERMINISTIC    
    COMMENT 'Returns the distance on the Earth between two known points of longitude and latitude'
    BEGIN
    DECLARE radius FLOAT;
    DECLARE distance FLOAT;

    IF unit = 'MILES' THEN SET radius = '3959';
    ELSEIF (unit = 'NAUTICAL_MILES' OR unit='NM') THEN SET radius = '3440.27694';   
    ELSEIF (unit = 'YARDS' OR unit='YD') THEN SET radius = '6967840';
    ELSEIF (unit = 'FEET' OR unit='FT') THEN SET radius = '20903520';
    ELSEIF (unit = 'KILOMETRES' OR unit='KILOMETERS' OR unit='KM') THEN SET radius = '6371.3929';
    ELSEIF (unit = 'METRES' OR UNIT='METERS' OR unit='M') THEN SET radius = '6371392.9';
    ELSE SET radius = '3959'; /* default to miles */
    END IF;

    SET distance = (radius * ACOS(COS(RADIANS(fromLatitude)) * COS(RADIANS(toLatitude)) * COS(RADIANS(toLongitude) - RADIANS(fromLongitude)) + SIN(RADIANS(fromLatitude)) * SIN(RADIANS(toLatitude))));

    RETURN distance;
    END$$

DELIMITER ;

Here's a set of test queries set to find the distance between the London Eye and Buckingham Palace, just for an example. Obviously normally you'd substitute the destination with fields from your database of geo-located 'things' that you want to compare with.

SET @milesModifier = 3959;

SET @myLat = 51.503228;
SET @myLong = -0.119703;

SET @destLat = 51.501267;  
SET @destLong = -0.142697;

SELECT  @kilometerModifier AS radius,
    @myLat AS myLat,
    @myLong AS myLong,
    @destLat AS destLat,
    @destLong AS destLong,
    (@milesModifier * ACOS(COS(RADIANS(@myLat)) * COS(RADIANS(@destLat)) * COS(RADIANS(@destLong) - RADIANS(@myLong)) + SIN(RADIANS(@myLat)) * SIN(RADIANS(@destLat)))) AS longFormat,
    haversine(@myLat,@myLong,@destLat,@destLong,'MILES') AS distanceMiles,
    haversine(@myLat,@myLong,@destLat,@destLong,'NAUTICAL_MILES') AS distanceNautical,
    haversine(@myLat,@myLong,@destLat,@destLong,'KM') AS distanceKm,
    haversine(@myLat,@myLong,@destLat,@destLong,'METRES') AS distanceMetres,    
    haversine(@myLat,@myLong,@destLat,@destLong,'YARDS') AS distanceYards,
    haversine(@myLat,@myLong,@destLat,@destLong,'FEET') AS distanceFeet,
    haversine(@myLat,@myLong,@destLat,@destLong,'') AS distanceDefault

In the example, we're using miles - so we've set the radius (@milesModifier in the test, radius in the function) to 3959 exactly.

The result I got back was interesting (on MySQL 5.2.6 community edition), highlights:

| longFormat       | distanceMiles   |
|------------------|-----------------|
| 0.99826000106148 | 0.9982578754425 |

longFormat is the maths done in the query, distanceMiles is the result of the function.

The results are different... OK, so its an insignificance as far as using the function in a project, but i'm interested to know how the same formula inside or outside of the function have different results.

I'm guessing that its to do with lengths of the FLOAT - they're not specified in the function, I have tried specifying them (right up to 30,15) to give plenty of room for all the figures I have and the output I expect - but the results still differ slightly.

Answer 1

FLOAT is an approximate data type - see:

Problems with Floating-Point Values
Numeric Types

Try changing FLOAT to DECIMAL(30,15) to ensure that you have the correct precision.

If you want an in-depth discussion of floating point, you could try this article:

What Every Computer Scientist Should Know About Floating-Point Arithmetic