Python cartesian product and conditions?

Question

In Python I am using the itertools.product() function to generate input parameters for a simulation.

I have a test function that requires 4 input parameters a1, a2, b1 and b2. I use the following code generate the parameters. Example:

params = itertools.product(range(10,41,2), range(10,41,2), range(0, 2), range(5, 31, 5))

… which gives me 3072 combinations. Unfortunately some combinations logically make no sense. E. g. if a2 is larger than a1 the test results are useless, also when b1 equals 0 the value of b2 is completely irrelevant – so it wouldn’t make sense to test such combinations.

Is there a possibility to restrict or filter the cartesian product beside doing it manually and nesting for-loops? Because my real use case has way more than 4 parameters, that’s why I like the convenience of the cartesian product function from itertools.

Any ideas or alternatives? Any help appreciated, thanks.

Answer 1

If you have many parameters, a constraint-based approach using a module like python-constraint may be easier to work with - let it do the hard work of figuring out which combinations are valid.

This would look something like

from constraint import Problem

prob = Problem()
prob.addVariables(["a1", "a2"], range(10,41,2))
prob.addVariable("b1", [0, 2])
prob.addVariable("b2", range(5, 31, 5))
prob.addConstraint(lambda a1, a2: a2 <= a1, ["a1", "a2"])
prob.addConstraint(lambda b1, b2: b1 != 0 or b2 == 5, ["b1", "b2"])

for params in prob.getSolutionIter():
    run_sim(**params)

Answer 2

You can use list comprehension or itertools.ifilter:

filtered_params = itertools.ifilter
    (lambda (a1, a2, b1, b2): a1 <= a2 and (b1 != 0 or b2 == 5), params)

Note that both of these versions loop and filter out under the hood. If you want to avoid that, you'll need to construct an improved algorithm that creates the tuples without the undesirables.

Answer 3

One option is to make params another generator which itself is fed from itertools.product.

For example:

params = (prod for prod in itertools.product(...) if prod[2] <= prod[1])

You could add anything after the if depending on what conditions are. For instance prod[2] <= prod[1] and prod[3] != 0 would check for the conditions you state in your question, letting through only the results that you need and discarding any products which failed the tests.

Answer 4

In cases such as this, it can be most convenient/intuitive/readable to express your rules using numpy's vector operations. E.g.:

import numpy as np

arr = np.array(list(params), dtype = [('a1',int),('a2',int),('b1',int),('b2',int)])
arr = arr[ arr['a2'] <= arr['a1'] ]
arr = arr[ arr['b1'] != 0 ]

Answer 5

You can use list comprehension in combination with any restrictions on the parameters you choose. I recommend putting your parameters in a set before doing this, to make sure there is no unnecessary code. I wouldn't happen in the case you mention above, but one does not always use range to generate parameter options.

For instance, here a list of tuple parameters are created, where it is only a valid combination if parameter 1 is larger than parameter 2 + 10:

acceptableParamCombinations = 
[ (p1,p2) for p1 in set(range(10,41,2)) for p2 in set(range(10,41,2)) if p1 > p2 + 10 ]