iterate python nested lists efficiently

Question

I am working on a network traffic monitor project in Python. Not that familiar with Python, so I am seeking help here.

In short, I am checking both in and out traffic, I wrote it this way:

for iter in ('in','out'):
        netdata = myhttp()
        print data

netdata is a list consisting of nested lists, its format is like this:

[ [t1,f1], [t2,f2], ...]

Here t represents the moment and f is the flow. However I just want to keep these f at this moment for both in and out, I wonder any way to get an efficient code.

After some search, I think I need to use create a list of traffic(2 elements), then use zip function to iterate both lists at the same time, but I have difficulty writing a correct one. Since my netdata is a very long list, efficiency is also very important.

If there is anything confusing, let me know, I will try to clarify. Thanks for help

Answer 1

Apart from minor fixes on your code (the issues raised by @Zero Piraeus), your question was probably answered here. A possible code to traverse a list of lists in N degree (a tree) is the following:

def traverse(item):
    try:
        for i in iter(item):
            for j in traverse(i):
                yield j
    except TypeError:
        yield item

Example:

l = [1, [2, 3], [4, 5, [[6, 7], 8], 9], 10]
print [i for i in traverse(l)]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

The key to make it work is recursion and the key to make it work efficiently is using a generator (the keyword yield gives the hint). The generator will iterate through your list of lists an returning to you item by item, without needing to copy data or create a whole new list (unless you consume the whole generator assigning the result to a list, like in my example)

Using iterators and generators can be strange concepts to understand (the keyword yield mainly). Checkout this great answer to fully understand them

Answer 2

The code you've shown doesn't make a great deal of sense. Here's what it does:

• Iterate through the sequence 'in', 'out', assigning each of those two strings in turn to the variable iter (masking the built-in function iter() in the process) on its two passes through the loop.

• Completely ignore the value of iter inside the loop.

• Assign the result of myhttp() to the variable netdata on each pass through the loop.

• Completely ignore the value of netdata, and instead attempt to print the undefined variable data on each pass through the loop.

It's possible, given the nested list you describe, that you want something like this:

for t, f in myhttp():
    print t
    print f
    # ... or whatever you want to do with those values.

Answer 3

When trying one the the other answers, the function was unable to recurse, and so I modified it to not recurse. It still works quite quickly, and can handle large nested lists (at least as far as I can tell with my testing). It is a Python 3 only function.

# Originally by Bruno Polaco
def traverse(item, reverse=False):
    its = [item] #stack of items to-be-processed
    out = [] # Output (no longer generator)
    ite = False
    while len(its) > 0:
        it = its.pop()
        try: # Check if item is iterable
            iter(it)
            ite = not isinstance(it, str)
        except TypeError:
            ite = False
        if ite: # Do something with it
            for i in it:
                its.append(i)
        else:
            out.append(it)
    if not reverse:
        out.reverse()
    return out