How deterministic is floating point inaccuracy?

Question

I understand that floating point calculations have accuracy issues and there are plenty of questions explaining why. My question is if I run the same calculation twice, can

Answer 1

I think your confusion lies in the type of inaccuracy around floating point. Most languages implement the IEEE floating point standard This standard lays out how individual bits within a float/double are used to produce a number. Typically a float consists of a four bytes, and a double eight bytes.

A mathmatical operation between two floating point numbers will have the same value every single time (as specified within the standard).

The inaccuracy comes in the precision. Consider an int vs a float. Both typically take up the same number of bytes (4). Yet the maximum value each number can store is wildly different.

• int: roughly 2 billion
• float: 3.40282347E38 (quite a bit larger)

The difference is in the middle. int, can represent every number between 0 and roughly 2 billion. Float however cannot. It can represent 2 billion values between 0 and 3.40282347E38. But that leaves a whole range of values that cannot be represented. If a math equation hits one of these values it will have to be rounded out to a representable value and is hence considered "inaccurate". Your definition of inaccurate may vary :).

Answer 2

Very few FPUs meet the IEEE standard (despite their claims). So running the same program in different hardware will indeed give you different results. The results are likely to be in corner cases that you should already avoid as part of using an FPU in your software.

IEEE bugs are often patched in software and are you sure that the operating system you are running today includes the proper traps and patches from the manufacturer? What about before or after the OS has an update? Are all bugs removed and bug fixes added? Is the C compiler in sync with all of this and is the C compiler producing the proper code?

Testing this may prove futile. You wont see the problem until you deliver the product.

Observe FP rule number 1: Never use an if(something==something) comparison. And rule number two IMO would have to do with ascii to fp or fp to ascii (printf, scanf, etc). There are more accuracy and bug problems there than in the hardware.

With each new generation of hardware (density) the affects from the sun are more apparent. We already have problems with SEU's on the planets surface, so independent of floating point calculations, you will have problems (few vendors have bothered to care, so expect crashes more often with new hardware).

By consuming enormous amounts of logic the fpu is likely to be a very fast (a single clock cycle). Not any slower than an integer alu. Do not confuse this with modern fpus being as simple as alus, fpus are expensive. (alus likewise consume more logic for multiply and divide to get that down to one clock cycle but its not nearly as big as the fpu).

Keep to the simple rules above, study floating point a bit more, understand the warts and traps that go with it. You may want to check for infinity or nans periodically. Your problems are more likely to be found in the compiler and operating system than the hardware (in general not just fp math). Modern hardware (and software) is, these days, by definition full of bugs, so just try to be less buggy than what your software runs on.

Answer 3

This is not a full answer to your question, but here is an example demonstrating that double calculations in C# are non-deterministic. I don't know why, but seemingly unrelated code can apparently affect the outcome of a downstream double calculation.

1. Create a new WPF application in Visual Studio Version 12.0.40629.00 Update 5, and accept all the default options.

2. Replace the contents of MainWindow.xaml.cs with this:

   using System;
   using System.Windows;
   
   namespace WpfApplication1
   {
       /// <summary>
       /// Interaction logic for MainWindow.xaml
       /// </summary>
       public partial class MainWindow : Window
       {
           public MainWindow()
           {
               InitializeComponent();
               Content = FooConverter.Convert(new Point(950, 500), new Point(850, 500));
           }
       }
   
       public static class FooConverter
       {
           public static string Convert(Point curIPJos, Point oppIJPos)
           {
               var ij = " Insulated Joint";
               var deltaX = oppIJPos.X - curIPJos.X;
               var deltaY = oppIJPos.Y - curIPJos.Y;
               var teta = Math.Atan2(deltaY, deltaX);
               string result;
               if (-Math.PI / 4 <= teta && teta <= Math.PI / 4)
                   result = "Left" + ij;
               else if (Math.PI / 4 < teta && teta <= Math.PI * 3 / 4)
                   result = "Top" + ij;
               else if (Math.PI * 3 / 4 < teta && teta <= Math.PI || -Math.PI <= teta && teta <= -Math.PI * 3 / 4)
                   result = "Right" + ij;
               else
                   result = "Bottom" + ij;
               return result;
           }
       }
   }
   

3. Set build configuration to "Release" and build, but do not run in Visual Studio.

4. Double-click the built exe to run it.
5. Note that the window shows "Bottom Insulated Joint".

6. Now add this line just before "string result":

   string debug = teta.ToString();
   

7. Repeat steps 3 and 4.

8. Note that the window shows "Right Insulated Joint".

This behavior was confirmed on a colleague's machine. Note that the window consistently shows "Right Insulated Joint" if any of the following are true: the exe is run from within Visual Studio, the exe was built using the Debug configuration, or "Prefer 32-bit" is unchecked in project properties.

It's quite difficult to figure out what's going on, since any attempt to observe the process appears to change the result.

Answer 4

HM. Since the OP asked for C#:

Is the C# bytecode JIT deterministic or does it generate different code between different runs? I don't know, but I wouldn't trust the Jit.

I could think of scenarios where the JIT has some quality of service features and decides to spend less time on optimization because the CPU is doing heavy number crunching somewhere else (think background DVD encoding)? This could lead to subtle differences that may result in huge differences later on.

Also if the JIT itself gets improved (maybe as part of a service pack maybe) the generated code will change for sure. The 80 bit internal precision issue has already been mentioned.