Incorrect VBA Overflow using Doubles (Excel)

假如想象 提交于 2019-12-11 03:19:23

问题


For some reason, the following statement evaluates to zero. I assume it's due to overflow, but all the interim values seem to be well within the limits of a double.

DiffieHellmanKey = (43 ^ 47) - (53 * Fix((43 ^ 47) / 53))

I think it's overflow because when I execute it with different numbers (below), it results in the correct value of 29.

DiffieHellmanKey = (5 ^ 22) - (53 * Fix((5 ^ 22) / 53))

What gives? Now, back to the original numbers that are giving me overflow. All variables involved are Doubles. It doesn't even work if I calculate it as a Worksheet formula instead of in VBA:

=(43 ^ 47) - (53 * ROUNDDOWN(((43 ^ 47) / 53), 0))

And if I implement the above example in VBA using the equivalent form (below), I get an incorrect result of -1.75357E+62.

DiffieHellmanKey = (43 ^ 47) - (53 * WorksheetFunction.RoundDown(((43 ^ 47) / 53), 0))

回答1:


You are kind of right but your problem is not with overflow, but with significant digits.

Numbers in Microsoft Excel can never have more than 15 significant digits, but decimals can be as large as 127.

source: http://msdn.microsoft.com/en-us/library/office/ff838588.aspx

This is what was happening with your original formula:

(43 ^ 47) - (53 * Fix((43 ^ 47) / 53)) simplifies to (43 ^ 47) - fix(43 ^ 47) Then the (43^47) has about 76 digits long so they are both getting cut down to the same amount and equating to 0.

The largest variable type in VBA is 'Decimal' Which only holds 29 digits of significance. You won't be able to perform math this large using visual basic natively.




回答2:


? (43^47) 
 5.92949097309764E+76

? 53*Fix((43^47)/53)
 5.92949097309764E+76 

? (53*Fix((43^47)/53)) = (43^47)
True


来源:https://stackoverflow.com/questions/12965805/incorrect-vba-overflow-using-doubles-excel

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!