Avoid overflow with softplus function in python

Question

I am trying to implement the following softplus function:

log(1 + exp(x))

I\'ve tried it with math/numpy and float64 as data type, but whenever

Answer 1

Since for x>30 we have log(1+exp(x)) ~= log(exp(x)) = x, a simple stable implementation is

def safe_softplus(x, limit=30):
  if x>limit:
    return x
  else:
    return np.log1p(np.exp(x))

In fact | log(1+exp(30)) - 30 | < 1e-10, so this implementation makes errors smaller than 1e-10 and never overflows. In particular for x=1000 the error of this approximation will be much smaller than float64 resolution, so it is impossible to even measure it on the computer.