deep-copying a generator in python

21
votes

I'm using a generator function, say:

def foo():
    i=0
    while (i<10):
         i+=1
         yield i

Now, I would like the option to copy the generator after any number of iterations, so that the new copy will retain the internal state (will have the same 'i' in the example) but will now be independent from the original (i.e. iterating over the copy should not change the original).

I've tried using copy.deepcopy but I get the error:

 "TypeError: object.__new__(generator) is not safe, use generator.__new__()"

Obviously, I could solve this using regular functions with counters for example. But I'm really looking for a solution using generators.

2
pythongenerator
I don't think it's possible. If all you need is two iterations over the results, then read to a list and iterate the list multiple times, or itertools.tee might help. It doesn't copy the generator, though, it just stores up results in a queue and spits them out again later. So any side-effects your function has won't be executed again when you read the "copy" (which presumably is what you'd want from a true clone of a generator), and if you read from the "copies" it will advance the underlying generator -- once you've teed you basically need all readers to use tees, not the original. - Steve Jessop
What is the problem you really want to solve? There are multiple possible answers. - Corley Brigman
You might know this already, but Define is not legal python. The correct keyword is def - SethMMorton
Nope, still wrong after the edit. Case is important in Python. Def is not legal either. Just def. - SethMMorton

2 Answers

11
votes

There are three cases I can think of:

  • Generator has no side effects, and you just want to be able to walk back through results you've already captured. You could consider a cached generator instead of a true generator. You can shared the cached generator around as well, and if any client walks to an item you haven't been to yet, it will advance. This is similar to the tee() method, but does the tee functionality in the generator/cache itself instead of requiring the client to do it.

  • Generator has side effects, but no history, and you want to be able to restart anywhere. Consider writing it as a coroutine, where you can pass in the value to start at any time.

  • Generator has side effects AND history, meaning that the state of the generator at G(x) depends on the results of G(x-1), and so you can't just pass x back into it to start anywhere. In this case, I think you'd need to be more specific about what you are trying to do, as the result depends not just on the generator, but on the state of other data. Probably, in this case, there is a better way to do it.

6
votes

The comment for itertools.tee was my first guess as well. Because of the warning that you shouldn't advance the original generator any longer after using tee, I might write something like this to spin off a copy:

>>> from itertools import tee
>>>
>>> def foo():
...   i = 0
...   while i < 10:
...     i += 1
...     yield i
...
>>>
>>> it = foo()
>>> it.next()
1
>>> it, other = tee(it)
>>> it.next()
2
>>> other.next()
2