logistic / sigmoid function implementation numerical precision

Question

in scipy.special.expit, logistic function is implemented like the following:

if x < 0
    a = exp(x) 
    a / (1 + a) 
else 
    1 / (1 + exp(-x))

However, I have seen implementations in other languages/frameworks that simply do

1 / (1 + exp(-x))

I am wondering how much benefit the scipy version actually brings.

For very small x, the result approaches to 0. It works even if exp(-x) overflows to Inf.

Answer 1

It's really just for stability - putting in values that are very large in magnitude might return unexpected results otherwise.

If expit was implemented just as 1 / (1 + exp(-x)) then putting a value of -710 into the function would return nan, whereas -709 would give a value close to zero as expected. This is because exp(710) is too big to be a double.

The branching in the code just means that this scenario is avoided.

See also this question and answer on Stack Overflow.

Answer 2

Seems it would be more efficient to use:

if x < -709
  sigmoid = 0.0
else
  sigmoid = 1.0 / (1.0 + exp(-x))

unless you need number with 10^-309 precision (see below) which seems overkill!

>>> 1 / (1 + math.exp(709.78))
5.5777796105262746e-309

Answer 3

Another way to do it would be

python np.where(x > 0, 1. / (1. + np.exp(-x)), np.exp(x) / (np.exp(x) + np.exp(0)))

Since np.exp(x) / (np.exp(x) + np.exp(0)) is equivalent to 1. / (1. + np.exp(-x)) but more stable for negative values