Parallel many dimensional optimization

Question

I am building a script that generates input data [parameters] for another program to calculate. I would like to optimize the resulting data. Previously I have been using the num

Answer 1

There are two ways of estimating gradients, one easily parallelizable, one not:

• around a single point, e.g. (f( x + h direction_i ) - f(x)) / h; this is easily parallelizable up to Ndim
• "walking" gradient: walk from x₀ in direction e₀ to x₁, then from x₁ in direction e₁ to x₂ ...; this is sequential.

Minimizers that use gradients are highly developed, powerful, converge quadratically (on smooth enough functions). The user-supplied gradient function can of course be a parallel-gradient-estimator.
A few minimizers use "walking" gradients, among them Powell's method, see Numerical Recipes p. 509.
So I'm confused: how do you parallelize its inner loop ?

I'd suggest scipy fmin_tnc with a parallel-gradient-estimator, maybe using central, not one-sided, differences.
(Fwiw, this compares some of the scipy no-derivative optimizers on two 10-d functions; ymmv.)