precision

问题 Split off from: https://stackoverflow.com/questions/31076846/is-it-possible-to-use-javascript-to-draw-an-svg-that-is-precise-to-1-000-000-000 Using floating point precision, there are rounding errors when calculating the position of points on a circle. Example: function pointsOnACircle (whichpoint, numberOfPoints, cx, cy, radius) { //Returns a point along the outside of a circle, starting at (whichpoint=1) = 0 degrees eachpoint = whichpoint * 2 * Math.PI /numberOfPoints; var thiscx = cx +

问题 I am encountering a floating point precision issue, does anybody know why this happens? Why is it that the cosine function is affected, but not he sine function. Math.sin(90 * Math.PI / 180); // returned: 1, expected: 1 1 - Math.sin(90 * Math.PI / 180); // returned: 0, expected: 0 Math.cos(90 * Math.PI / 180); // returned: 6.123233995736766e-17, expected: 0 1 - Math.cos(90 * Math.PI / 180); // returned: 0.9999999999999999, expected: 1 回答1: The canonical answer to this one is What Every

问题 So this the line with the precision error fault; A[i]= m % 3; m is long A is int[]; And my error is error: possible loss of precision A[i]= m % 3. required int found long. How can I have error when the only potential answers are 0,1,2? Isn't there another way than declaring A as long[]? It's a potentially big array so I don't want that (in fact I would even prefer for A to be short[]) Also I tried error: A[i]= m % 3L , but same result. 回答1: The compiler doesn't look at the result, it looks at

问题 This question already has answers here : Is There a Better Double-Precision Assignment in Fortran 90? (2 answers) Closed 2 years ago . When I run the code below I get an output of 6378136 .5 instead of 6378136 .3 PROGRAM test implicit none real*8 radius radius = 6378136.3 print*,radius END I have read this other link (Precision problems of real numbers in Fortran) but it doesn't explain how to fix this problem. 回答1: The reason this is happening is not because the variable you are using lacks

问题 I am trying to perform an addition on a double atomically in a loop using a compare and exchange scheme with this function: namespace my { template<typename value_type> value_type atomic_add(std::atomic<value_type>& operand, value_type value_to_add) { value_type old = operand.load(std::memory_order_consume); value_type desired = old + value_to_add; while (!operand.compare_exchange_weak(old, desired, std::memory_order_release, std::memory_order_consume)) desired = old + value_to_add; return

问题 MySQL documentation says : The DECIMAL and NUMERIC types store exact numeric data values. These types are used when it is important to preserve exact precision, for example with monetary data. I did this test on that column decimal_column DECIMAL(31,30) . insert into tests (decimal_column) values(1/3); then inspecting what has been stored gives this select * from tests ; > result : 0.333333333000000000000000000000 then reversing the math operation with this query gives this select decimal

问题 This is somewhat related to the answer. I understand the float is not suggested for high precision. But I don't exactly get what's going on with FLOAT after going through the below results, Observation 1 The value returned by the below 2 queries returns different results, SELECT MY_FLOAT_COL FROM MY_TABLE; SELECT MY_FLOAT_COL*1 FROM MY_TABLE; The possible explanation is the reference in the above link, which says the float is approximate value and the problem is the datatype. Observation 2

问题 I'm calculating small angle between large vectors in 3D. I use common formula: double angle = atan2(norm(cross_product), dot_product); In general I'm trying to get angle between my vector on Earth and Normal to Earth elipsoid(gravitational vector) to align my vector to gravitational vector. I store my vector in ECEF coordinates. And values are about: normal_ecef[2853169.5286430484, 2139463.4659523461, 5254245.1411450412] v_ecef[11.4893763275, 9.3040978122, 16.5129716340] cross_product

问题 I'm trying to get position of a fragment and change the color at positions, but when I try to do it, I get a weird effect where they don't want to be changed completely. Notice how there is a cyan color, but that part should be transparent. It seems like there's some weird rounding happening, and I'm not sure how to fix it. What is happening here? void main() { vec4 c0 = vec4(0,1,0,1); c0.a *= step(132.0,gl_FragCoord.x); gl_FragColor = c0; } 回答1: You need to show more code. Did you enable

问题 I have a data file with 3 columns and I'd like to plot a 2D map plot. What I do in gnuplot is: p'datafile' u 1:2:3 with image for some set of data, the data on the 3rd column (say Z) are different with order of 0.01, e.g 1.56, 1.58, 1.59 etc. I'd like to skip those small difference and consider them all 1.5. How can I set gnuplot to consider only up to one digit after decimal for 3rd column? Thanks! 回答1: You can use the floor function to round your numbers down: plot 'datafile' using 1:2: