TSQL number rounding issue

问题

I have a piece of code:

IF OBJECT_ID(N'dbo.rounding_testing') IS NOT NULL
    DROP FUNCTION dbo.rounding_testing;
GO
CREATE FUNCTION dbo.rounding_testing
(
    @value FLOAT,
    @digit INT
)
RETURNS FLOAT
BEGIN
    DECLARE
        @factor FLOAT,
        @result FLOAT;
    SELECT @factor = POWER(10, @digit);

    SELECT @result = FLOOR(@value * @factor + 0.4);

    RETURN @result;
END;
GO

SELECT dbo.rounding_testing(5.7456, 3);
SELECT FLOOR(5.7456 * 1000 + 0.4);

The results are:

5745
5746

I'm expecting two 5746. I tried to debug the function and found some interesting behavior. Below are some testing I did in the Immediate Window when debugging.

@factor
1.000000000000000e+003
@result
5.745000000000000e+003
@value
5.745600000000000e+000
@value*@factor
5745.6
@value*@factor+0.4
5746
floor(@value*@factor+0.4)
5745
floor(5746)
5746

Can anyone help to explain the result? Especially these three lines:

@value*@factor+0.4
5746
floor(@value*@factor+0.4)
5745
floor(5746)
5746

回答1:

In the expression FLOOR(5.7456 * 1000 + 0.4);, the part between parentheses is evaluated first. For constants the data types are inferred based on the notation; for 5.7456 that is decimal(5,4); 1000 is an int; and 0.4 is decimal(1,1). The inferred data type for 5.7456 * 1000 is then decimal(10,4); and for the full expression it is decimal(11,4). These are all exact numeric data types so you will not experience any rounding; the end result is 5746.0000 exactly. The FLOOR function trims the fraction and converts to decimal(11,0), returning 5746.

In the user-defined function, you store input parameters and intermediate results in float data type (floating point data). This data type is intended to be used for approximate data, such as measurements, where the data you read from the intstrument is already an approximation. I have learned in high school to read as many digits as I can, but treat the last one as insignificant - I had to keep it in all computations, but round the final result to the number of significant digits based on the accuracy of my measurements. The rounding ensures that inaccuracies in the last digits will not affect the end result. Floating point data types should be treated in the same way.

Internally, floating point digits are represented in a base-2 number system. This means that there are numbers that have an exact representation in our commonly used base-10 system (such as 5.7456), but a never ending fractional part in base-2. (Similar to how for instance one third, which can be represented exactly in base-3, has a never ending fractional part in base-10: 0.33333333333(etc)). The number of base-2 digits used for storage of a float number is finite, so it has to be cut off at the end - which results in it being rounded either up or down by a tiny fraction. You can see this if you run:

DECLARE @a float = 5.7456;
SELECT CAST(@a AS decimal(19,16));

In this case, the effect of cutting off after a lot of base-2 digits is that the value stored is 0.0000000000000004 less than the decimal value you put in. That small difference turns into a huge effect because of the FLOOR function, which does exactly what it should do: round down to the nearest integer.

(I've seen a lot of people call this an error. It is not. It is intended and documented behavior. And the precision loss here is neither worse nor better than the precision loss you get when you store a third in a DECIMAL(7,6); it is just a bit less obvious because we have all grown up being used to working in base-10)

回答2:

You issue can be fixed by changing float to real

IF OBJECT_ID(N'dbo.rounding_testing') IS NOT NULL
    DROP FUNCTION dbo.rounding_testing;
GO
CREATE FUNCTION dbo.rounding_testing
(
    @value REAL,
    @digit INT
)
RETURNS REAL
BEGIN
    DECLARE
        @factor REAL,
        @result REAL;
    SELECT @factor = POWER(10, @digit);

    SELECT @result = FLOOR(@value * @factor + 0.4);

    RETURN @result;
END;
GO

SELECT dbo.rounding_testing(5.7456, 3);
SELECT FLOOR(5.7456 * 1000 + 0.4);

来源：https://stackoverflow.com/questions/41178630/tsql-number-rounding-issue

标签

sql-server

floating-point

rounding

floor