How do I efficiently filter computed values within a Python list comprehension?

Question

The Python list comprehension syntax makes it easy to filter values within a comprehension. For example:

result = [x**2 for x in mylist if type(x) is int]
<         


        

Answer 1

I will have a preference for:

itertools.ifilter(bool, (expensive(x) for x in mylist))

This has the advantage to:

• avoid None as the function (will be eliminated in Python 3): http://bugs.python.org/issue2186
• use only iterators.

Answer 2

This is exactly what generators are suited to handle:

result = (expensive(x) for x in mylist)
result = (do_something(x) for x in result if some_condition(x))
...
result = [x for x in result if x]  # finally, a list

1. This makes it totally clear what is happening during each stage of the pipeline.
2. Explicit over implicit
3. Uses generators everywhere until the final step, so no large intermediate lists

cf: 'Generator Tricks for System Programmers' by David Beazley

Answer 3

There is the plain old use of a for loop to append to a list, too:

result = []
for x in mylist:
    expense = expensive(x)
    if expense:
        result.append(expense)

Answer 4

Came up with my own answer after a minute of thought. It can be done with nested comprehensions:

result = [y for y in (expensive(x) for x in mylist) if y]

I guess that works, though I find nested comprehensions are only marginally readable

Answer 5

result = [x for x in map(expensive,mylist) if x]

map() will return a list of the values of each object in mylist passed to expensive(). Then you can list-comprehend that, and discard unnecessary values.

This is somewhat like a nested comprehension, but should be faster (since the python interpreter can optimize it fairly easily).

Answer 6

You could memoize expensive(x) (and if you are calling expensive(x) frequently, you probably should memoize it any way. This page gives an implementation of memoize for python:

http://code.activestate.com/recipes/52201/

This has the added benefit that expensive(x) may be run less than N times, since any duplicate entries will make use of the memo from the previous execution.

Note that this assumes expensive(x) is a true function, and does not depend on external state that may change. If expensive(x) does depend on external state, and you can detect when that state changes, or you know it wont change during your list comprehension, then you can reset the memos before the comprehension.