Logistic regression using SciPy

Question

I am trying to code up logistic regression in Python using the SciPy fmin_bfgs function, but am running into some issues. I wrote functions for the logistic (si

Answer 1

Your problem is that the function you are trying to minimise, logLikelihoodLogit, will return NaN with values very close to your initial estimate. And it will also try to evaluate negative logarithms and encounter other problems. fmin_bfgs doesn't know about this, will try to evaluate the function for such values and run into trouble.

I suggest using a bounded optimisation instead. You can use scipy's optimize.fmin_l_bfgs_b for this. It uses a similar algorithm to fmin_bfgs, but it supports bounds in the parameter space. You call it similarly, just add a bounds keyword. Here's a simple example on how you'd call fmin_l_bfgs_b:

from scipy.optimize import fmin_bfgs, fmin_l_bfgs_b

# list of bounds: each item is a tuple with the (lower, upper) bounds
bd = [(0, 1.), ...] 

test = fmin_l_bfgs_b(logLikelihoodLogit, x0=x0, args=(mX, vY), bounds=bd,
                      approx_grad=True)

Here I'm using an approximate gradient (seemed to work fine with your data), but you can pass fprime as in your example (I don't have time to check its correctness). You'll know your parameter space better than me, just make sure to build the bounds array for all the meaningful values that your parameters can take.