precision

问题 My question, in short is, why does rounding error in floats only show up after calculations and not for storage of literals? What I mean is this - I know about the issues that arise due to rounding error in floats when converting from decimal to binary and back. Eg, in Java: double a = 10.567; double b = 2.16; double c = a * b; c then stores the value 22.824720000000003, instead of 22.82472. This is because the result 22.82472 cannot be stored accurately in the finite binary digits of the

问题 According to the entry for decimal and numeric data types in SQL Server 2008 Books Online, precision is: p (precision) The maximum total number of decimal digits that can be stored, both to the left and to the right of the decimal point. The precision must be a value from 1 through the maximum precision of 38. The default precision is 18. However, the second select below fails with "Arithmetic overflow error converting int to data type numeric." SELECT CAST(123456789 as decimal(9,0)) SELECT

问题 Using SQL Server 2012... I have two columns: Price [decimal(28,12)] OustandingShares [decimal(38,3)] -- The 38 is overkill but alas, not my call. When I do an ALTER TABLE I get a resulting computed column as a [decimal(38,6)]. I need the datatype to be [decimal(28,12)]. ALTER TABLE [xyz].MyTable ADD Mv AS OustandingShares * Price How can I effectively get 12 decimals of scale on this computed column? I've tried doing convert on the OutstandingShares to 12 decimal places as well as wrapping a

问题 I'm building a program to to convert double values in to scientific value format(mantissa, exponent). Then I noticed the below 369.7900000000000 -> 3.6978999999999997428 68600000 -> 6.8599999999999994316 I noticed the same pattern for several other values also. The maximum fractional error is 0.000 000 000 000 001 = 1*e-15 I know the inaccuracy in representing double values in a computer. Can this be concluded that the maximum fractional error we would get is 1*e-15 ? What is significant

问题 In a C code (Linux OS), I need to sleep accurately inside a while loop - say, 10000 microseconds for 1000 times. I have tried usleep, nanosleep, select, pselect, and some other methods with no success. Once in ~50 times it would sleep %100 longer (~20000 us). I need to perform an action after each delay. So, each delay has to be very accurate. Is there a way to do accurate sleeps for this case? Thanks.. EDIT: #include <stdio.h> #include <sys/time.h> int main(int argc, char *argv[]) { struct

问题 In javascipt I have av floating point "a" like this: var a = 5.; Now I want a new number "b" that is barely larger than "a". I could do this: var b = a + 1.e-10; But what if "a" is a really small number? var a = 5.e-20; var b = a + 1.e-10; Now "b" is many orders of magnitude larger than "a". Also if I make the difference between "a" and "b" too small, a large "a" could cause the difference to be rounded off. How do I make the number "b" larger than any number "a", but closer to "a" than any

问题 I am trying to establish some concise overview of what options for precise caluclations we have in JAVA+SQL. So far I have found following options: use doubles accepting their drawbacks, no go. use BigDecimals using them in complicated formulas is problematic for me use String.format/Decimal.format to round doubles do i need to round each variable in formula or just result to get BigDecimal precision? how can this be tweaked? use computed fields option in SQL. drawback is that I'd need

问题 What are the smallest positive values in C++ for which their multiplicative inverse will still be finite? Have tried numeric_limits<double>::epsilon() but it isn't it - am getting positive values for much smaller values than that. #include <limits>//is it here? void tfuuuuu() { double d_eps,invd_eps; float f_eps,invf_eps; invd_eps = 1.0/d_eps;//invd_eps should be finite invf_eps = 1.f/f_eps;//invf_eps should be finite } 回答1: I doubt there is a standard library function for finding the number

问题 I was trying to answer a question from here (Substracion of two items from two lists). The original problem has two different lists with float values and the goal is to zip them and substract This code works fine: Enter=[7.12, 7.14, 7.24, 7.45, 7.28, 7.31, 7.18, 7.25, 7.33, 7.38] Leave=[7.56, 7.24, 7.48, 7.52, 7.45, 7.57, 7.22, 7.31, 7.37, 7.41] intervals = map(lambda x, y: y-x, Enter, Leave) Then: print intervals Output: [0.4399999999999995, 0.10000000000000053, 0.24000000000000021, 0

问题 I have created a double-double data type in C. I tried -Ofast with GCC and discovered that it's dramatically faster (e.g. 1.5 s with -O3 and 0.3s with -Ofast ) but the results are bogus. I chased this down to -fassociative-math . I'm surprised this does not work because I explicitly define the associativity of my operations when it matters. For example in the following code I but parentheses where it matters. static inline doublefloat two_sum(const float a, const float b) { float s = a + b;