OverflowError: long int too large to convert to float in python

Question

I tried to calculate poisson distribution in python as below:

p = math.pow(3,idx)
depart = math.exp(-3) * p 
depart = depart / math.factorial(idx)
         


        

Answer 1

When idx gets large either the math.pow and/or the math.factorial will become insanely large and be unable to convert to a floating value (idx=1000 triggers the error on my 64 bit machine). You'll want to not use the math.pow function as it overflows earlier than the built in ** operator because it tries to keep higher precision by float converting earlier. Additionally, you can wrap each function call in a Decimal object for higher precision.

Another approach when dealing with very large numbers is to work in the log scale. Take the log of every value (or calculate the log version of each value) and perform all required operations before taking the exponentiation of the results. This allows for your values to temporary leave the floating domain space while still accurately computing a final answer that lies within floating domain.

3 ** idx  =>  math.log(3) * idx
math.exp(-3) * p  =>  -3 + math.log(p)
math.factorial(idx)  =>  sum(math.log(ii) for ii in range(1, idx + 1))
...
math.exp(result)

This stays in the log domain until the very end so your numbers can get very, very large before you'll hit overflow problems.