Positive directional derivative for linesearch

Question

What does the smode of scipy.optimize \'Positive directional derivative for linesearch\' mean?

for example in fmin_slsqp http://docs.scipy.org/doc/scipy/reference/ge

Answer 1

To avoid changing your function you can also try experimenting with the ftol and eps parameters. Changing ftol to a higher value is equivalent to changing the function to a smaller value.

Answer 2

One situation in which you receive this error, is when

1. x0 is outside the valid range you defined in bounds.
2. and the unconstrained maximum is attained for values outside bounds.

I will set up a hypothetical optimization problem, run it with two different initial values and print the output of scipy.optimize:

import numpy as np
from scipy import optimize

H = np.array([[2., 0.],
              [0., 8.]])

c = np.array([0, -32])

x0 = np.array([0.5, 0.5])    # valid initial value
x1 = np.array([-1, 1.1])     # invalid initial value

def loss(x, sign=1.):
    return sign * (0.5 * np.dot(x.T, np.dot(H, x)) + np.dot(c, x))

def jac(x, sign=1.):
    return sign * (np.dot(x.T, H) + c)

bounds = [(0, 1), (0, 1)]

Now that loss function, gradient, x0 and bounds are in place, we can solve the problem:

def solve(start):
    res = optimize.minimize(fun=loss, 
                            x0=start, 
                            jac=jac, 
                            bounds=bounds,
                            method='SLSQP')
    return res



solve(x0)   # valid initial value
# fun: -27.999999999963507
# jac: array([ 2.90878432e-14, -2.40000000e+01])
# message: 'Optimization terminated successfully.'
# ...
#  status: 0
# success: True
# x: array([1.45439216e-14, 1.00000000e+00])

solve(x1)      # invalid initial value:
#  fun: -29.534653465326528
#  jac: array([ -1.16831683, -23.36633663])
#  message: 'Positive directional derivative for linesearch'
#  ...
#  status: 8
#  success: False
#  x: array([-0.58415842,  1.07920792])

As @pv. pointed out in the accepted answer, the algorithm can't verify that this is a minimum:

I think this message means that the optimizer got into a position where it did not manage to find a direction where the value of the objective function decreases (fast enough), but could also not verify that the current position is a minimum.

Answer 3

I still don't know what it means but how to solve it. Basically, the function that is optimized needs to return a smaller value.

F(x):
    ...
    return value / 10000000

Answer 4

It's not a complete answer, but you can see the source code that generates the smode here:

https://github.com/scipy/scipy/blob/master/scipy/optimize/slsqp/slsqp_optmz.f

Assignments of mode = 8 (the "Positive directional derivative for linesearch" you are asking about) can be found in lines 412 and 486. If can figure out why they are assigned in the code, you've got your answer.

Answer 5

These optimization algorithms typically work by choosing a descent direction, and then performing a line search to that direction. I think this message means that the optimizer got into a position where it did not manage to find a direction where the value of the objective function decreases (fast enough), but could also not verify that the current position is a minimum.