Modifying a list while iterating over it - why not?

Question

Almost every tutorial and SO answer on this topic insists that you should never modify a list while iterating over it, but I can\'t see why this is such a bad thing if the c

Answer 1

I'll go into a little bit more detail why you shouldn't iterate over a list. Naturally, by that I mean

for elt in my_list:
    my_list.pop()

or similar idioms.

First, we need to think about what Python's for loop does. Since you can attempt to iterate over any object, Python doesn't necessarily know how to iterate over whatever you've given it. So there is a list (heh) of things it tries to do to work out how to present the values one-by-one. And the first thing it does is checks for an __iter__ method on the object and -- if it exists -- calls it.

The result of this call will then be an iterable object; that is, one with a next method. Now we're good to go: just call next repeatedly until StopIteration is raised.

Why is this important? Well, because the __iter__ method has actually to look at the data structure to find the values, and remember some internal state so that it knows where to look next. But if you change the data structure then __iter__ has no way of knowing that you've been fiddling, so it will blithely keep on trying to grab new data. What this means in practise is that you will probably skip elements of the list.

---

It's always nice to justify this sort of claim with a look at the source code. From listobject.c:

static PyObject *
listiter_next(listiterobject *it)
{
    PyListObject *seq;
    PyObject *item;

    assert(it != NULL);
    seq = it->it_seq;
    if (seq == NULL)
        return NULL;
    assert(PyList_Check(seq));

    if (it->it_index < PyList_GET_SIZE(seq)) {
        item = PyList_GET_ITEM(seq, it->it_index);
        ++it->it_index;
        Py_INCREF(item);
        return item;
    }

    Py_DECREF(seq);
    it->it_seq = NULL;
    return NULL;
}

Note in particular that it really does simulate a C-style for loop, with it->it_index playing the part of the index variable. In particular, if you delete an item from the list then you won't update it_index, so you may skip a value.