cocoa - I've discovered what I think is a bug with NSDecimalNumber

Question

Here is a simple code that shows what I think is a bug when dealing with double numbers...

double wtf = 36.76662445068359375000;
id xxx = [NSDecimalNumber nu         


        

Answer 1

Trying to go beyond roughly 16 digits of decimal precision with a double is generally not recommended. Frankly, I'm surprised that you are able to represent this double (with 19 significant digits) in a way that maintains that precision when logging it out. You may even get different behavior on the iPhone itself, which maps a long double type to just plain double (your Mac may be handling this as a long double behind the scenes).

The rounding you are seeing may be happening at the binary level (see here for more on this), so you won't be seeing the decimal rounding that you are expecting.

It is for these reasons that you will want to work entirely with NSDecimalNumbers or NSDecimals from start to finish if you need this kind of high-precision math. To do this, do not convert to and from floating point types, but instead use NSStrings directly to populate and export the numbers (or store them as NSDecimalNumbers in Core Data).

For example, you could work around the above problems with the following code:

id xxx = [NSDecimalNumber decimalNumberWithString:@"36.76662445068359375000"];

NSDecimalNumbers (and their C struct equivalent NSDecimal) can handle up to 38 significant digits.

Answer 2

Normally, I'm the one who comes in and explains to people that the number they entered is not representable as a floating-point number, and where the rounding errors are, blah blah blah.

This question is much more fun than what we usually see, and it illustrates exactly what's wrong with the crowd wisdom of "floating point is inexact, read 'what every computer scientist should know...' lolz".

36.76662445068359375 is not just any 19-digit decimal number. It happens to be a 19 digit decimal number that is also exactly representable in double precision binary floating point. Thus, the initial conversion implicit in:

double wtf = 36.76662445068359375000;

is exact. wtf contains exactly b100100.11000100010000011, and no rounding has occurred.

The spec for NSDecimalNumber says that it represents numbers as a 38 digit decimal mantissa and a decimal exponent in the range [-127,128], so the value in wtf is also exactly representable as an NSDecimalNumber. Thus, we may conclude that numberWithDouble is not delivering a correct conversion. Although I cannot find documentation that claims that this conversion routine is correctly rounded, there is no good reason for it not to be. This is a real bug, please report it.

I note that the string formatters on iPhoneOS seem to deliver correctly rounded results, so you can probably work around this by first formatting the double as a string with 38 digit precision and then using decimalNumberWithString. Not ideal, but it may work for you.