python - Why can't I iterate twice over the same data?

54
votes

Honestly I am a little confused here, why can't I iterate twice over the same data?

def _view(self,dbName):
    db = self.dictDatabases[dbName]
    data = db[3]

    for row in data:
        print("doing this one time")

    for row in data:
        print("doing this two times")

This will print out "doing this one time" a few times (as data has a few rows), however it will NOT print out "doing this two times" at all ...

The first time I iterate over data works fine, but the second time when I run the last list "for row in data" this returns nothing ... so executing it one time works but not twice ... ?

FYI - data is a csv.reader object (in case that is the reason)...

3
pythoniterator
Iterable vs. iterator. - Ignacio Vazquez-Abrams
I'm not saying that this is a duplicate, but you might also want to refer to stackoverflow.com/questions/9884132/… for some more context / explanation - Nick Meyer
Why don't you just iterate over the data once and do everything you need the first time? - nhgrif
Related: Resetting an iterator object - Aran-Fey

3 Answers

49
votes

It's because data is an iterator, an you can consume an iterator only once. For example:

lst = [1, 2, 3]
it = iter(lst)

next(it)
=> 1
next(it)
=> 2
next(it)
=> 3
next(it)
=> StopIteration

If we are traversing some data using a for loop, that last StopIteration will cause it to exit the first time. If we try to iterate over it again, we'll keep getting the StopIteration exception, because the iterator has already been consumed.

Now for the second question: What if we do need to traverse the iterator more than once? A simple solution would be to create a list with the elements, and we can traverse it as many times as needed. This is all right as long as there are few elements in the list:

data = list(db[3])

But if there are many elements, it's a better idea to create independent iterators using tee():

import itertools
it1, it2 = itertools.tee(db[3], n=2) # create as many as needed

Now we can loop over each one in turn:

for e in it1:
    print("doing this one time")

for e in it2:
    print("doing this two times")
10
votes

Once an iterator is exhausted, it will not yield any more.

>>> it = iter([3, 1, 2])
>>> for x in it: print(x)
...
3
1
2
>>> for x in it: print(x)
...
>>>
4
votes

I want to complete the answer of @ÓscarLópez for them who looks for a solution in 2017 and uses python 2.7 or 3.

Method tee() takes no keyword arguments now and waits for the second argument an integer, not keyword. This is the right way to use tee():

import itertools
it1, it2 = itertools.tee(db[3], 2)