How to print the progress of a list comprehension in python?

Question

In my method i have to return a list within a list. I would like to have a list comprehension, because of the performance since the list takes about 5 minutes to create.

Answer 1

def show_progress(it, milestones=1):
    for i, x in enumerate(it):
        yield x
        processed = i + 1
        if processed % milestones == 0:
            print('Processed %s elements' % processed)

Simply apply this function to anything you're iterating over. It doesn't matter if you use a loop or list comprehension and it's easy to use anywhere with almost no code changes. For example:

doc_collection = [[1, 2],
                  [3, 4],
                  [5, 6]]

result = [[str(token) for token in document]
          for document in show_progress(doc_collection)]

print(result)  # [['1', '2'], ['3', '4'], ['5', '6']]

If you only wanted to show progress for every 100 documents, write:

show_progress(doc_collection, 100)