How to use nested loops in joblib library in python

问题

Actual code looks like:

def compute_score(row_list,column_list): 

    for i in range(len(row_list)):
            for j in range(len(column_list)):
                tf_score = self.compute_tf(column_list[j],row_list[i])

I am tying to achieve multi-processing i.e. at every iteration of j I want to pool column_list. Since compute_tf function is slow I want to multi-process it.

I've found have to do it using joblib in Python, But I am unable to workaround with nested loops.

Parallel(n_jobs=2)(delayed(self.compute_tf)<some_way_to_use_nested_loops>)

This is what is to be achieved. It would be a great help if any solution on this is provided or any-other solution.

回答1:

Another solution without having to implement a generator function, is to use the nested list comprehension for the generator:

Parallel(n_jobs=2)(delayed(self.compute_tf)(i, j) for j in column_list for i in row_list)

The order will be given as:

[(i, j) for j in range(10) for i in range(10)]

回答2:

I'm not understanding your part about Parallel. However, your code can be simplified to this:

def compute_score(row_list,column_list): 

    for i in row_list:
            for j in column_list:
                tf_score = self.compute_tf(j,i)

It may be useful to create a generator that yields each j,i so that you can process them outside of the generator

def compute_pairs(row_list,column_list): 

    for i in row_list:
            for j in column_list:
                yield j, i

then use:

 scored = [compute_tf(j,i) for j, i in compute_pairs(row_list, column_list)]

来源：https://stackoverflow.com/questions/36423975/how-to-use-nested-loops-in-joblib-library-in-python

标签

python

multiprocessing

joblib