floating-accuracy

问题 >>> float(str(0.65000000000000002)) 0.65000000000000002 >>> float(str(0.47000000000000003)) 0.46999999999999997 ??? What is going on here? How do I convert 0.47000000000000003 to string and the resultant value back to float? I am using Python 2.5.4 on Windows. 回答1: str(0.47000000000000003) give '0.47' and float('0.47') can be 0.46999999999999997 . This is due to the way floating point number are represented (see this wikipedia article) Note: float(repr(0.47000000000000003)) or eval(repr(0

问题 I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I don\'t need the value rounded off. Hope this is possible in javascript. 回答1: upd : So, after all it turned out, rounding bugs will always haunt you, no matter how hard you try to compensate them. Hence the problem should be attacked by representing numbers exactly in decimal notation. Number.prototype

问题 I was wondering if there is a way of overcoming an accuracy problem that seems to be the result of my machine\'s internal representation of floating-point numbers: For the sake of clarity the problem is summarized as: // str is \"4.600\"; atof( str ) is 4.5999999999999996 double mw = atof( str ) // The variables used in the columns calculation below are: // // mw = 4.5999999999999996 // p = 0.2 // g = 0.2 // h = 1 (integer) int columns = (int) ( ( mw - ( h * 11 * p ) ) / ( ( h * 11 * p ) + g

问题 Just for fun and because it was really easy, I\'ve written a short program to generate Grafting numbers, but because of floating point precision issues it\'s not finding some of the larger examples. def isGrafting(a): for i in xrange(1, int(ceil(log10(a))) + 2): if a == floor((sqrt(a) * 10**(i-1)) % 10**int(ceil(log10(a)))): return 1 a = 0 while(1): if (isGrafting(a)): print \"%d %.15f\" % (a, sqrt(a)) a += 1 This code misses at least one known Grafting number. 9999999998 => 99999

问题 I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid. This is the core function for mine atmospheric scattering shader. The old original shader was on floats and for normal rendering was fine, but after addition of zoom I found that with relatively small distances the precision get lost. On floats the usable distances for Earth was only up to 0.005 AU (Astronomical Unit

问题 I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute: for ( float value = 0.0f; value < 1.0f; value += 0.1f ) System.out.println( value ); What I get is: 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.70000005 0.8000001 0.9000001 I understand that there is an accumulation of the floating precision error, however, how to get rid of this? I tried using doubles to half the error, but the result is still the same. Any ideas? 回答1: There is a no exact

问题 The following code is obviously wrong. What\'s the problem? i <- 0.1 i <- i + 0.05 i ## [1] 0.15 if(i==0.15) cat(\"i equals 0.15\") else cat(\"i does not equal 0.15\") ## i does not equal 0.15 回答1: General (language agnostic) reason Since not all numbers can be represented exactly in IEEE floating point arithmetic (the standard that almost all computers use to represent decimal numbers and do math with them), you will not always get what you expected. This is especially true because some

问题 Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen? 回答1: Binary floating point math is like this. In most programming languages, it is based on the IEEE 754 standard. JavaScript uses 64-bit floating point representation, which is the same as Java's double . The crux of the problem is that numbers are represented in this format as a whole number times a power of two; rational numbers (such as 0.1 , which is 1/10 ) whose

问题 This question already has an answer here: How to deal with floating point number precision in JavaScript? 40 answers I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying floating point precision? For example: 0.2 + 0.4 = 0.6000000000000001 Two things I have tried: >>> y = 1.23456789 >>> x = 0.2 >>> parseInt(Math.round(Math.floor(y/x))) * x; 1

问题 I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x == 0) { // do important operation } Isn\'t CGFloat vulnerable to false positives when comparing with == , <= , >= , < , > ? It is a floating point and they have unprecision problems: 0.0000000000041 for example. Is Objective-C handling this internally when comparing or can it happen that a origin