Python itertools.combinations() memory problems

Question

I'm processing a huge number of combinations of items (from League of Legends), about 72 million, all of which are fed into a function that calculates how beneficial they are.

We're trying to find the best possible combination.

Ignoring the fact that there might be better ways, algorithmically speaking, to do this, can anyone tell me why I'm getting a memory error?

allpossiblei = itertools.combinations(items.keys(),5)
maxc = 0
i = 0
for combo in allpossiblei:
    icombo = [items[name] for name in combo]
    res, tcost = calcStats(icombo, 0.658,100,100)
    if res > maxc :
        maxc = res
        print str(res) + " " + str(res/tcost)
        print combo
        print float(i)/79208745.0
    if i % 500000 == 0:
        print str(float(i)/79208745.0) + "\n \n"
        gc.collect()
    i = i + 1

calcStats doesn't do anything except arithmetic using local variables.

This rapidly eats up 2gb+ of memory and quits in about 5 minutes. I thought itertools was supposed to provide a generator that wouldn't use up a ton of memory? I even threw in that gc.collect() statement but it doesn't seem to work. Any ideas?

Answer 1

Combinations creates a pool by consuming the whole iterator provided. There is no way around that. See the pseudocode in the docs for the function: http://docs.python.org/library/itertools.html#itertools.combinations