The word "yield" has two meanings: to produce something (e.g., to yield corn), and to halt to let someone/thing else continue (e.g., cars yielding to pedestrians). Both definitions apply to Python's yield keyword; what makes generator functions special is that unlike in regular functions, values can be "returned" to the caller while merely pausing, not terminating, a generator function.
It is easiest to imagine a generator as one end of a bidirectional pipe with a "left" end and a "right" end; this pipe is the medium over which values are sent between the generator itself and the generator function's body. Each end of the pipe has two operations: push, which sends a value and blocks until the other end of the pipe pulls the value, and returns nothing; and pull, which blocks until the other end of the pipe pushes a value, and returns the pushed value. At runtime, execution bounces back and forth between the contexts on either side of the pipe -- each side runs until it sends a value to the other side, at which point it halts, lets the other side run, and waits for a value in return, at which point the other side halts and it resumes. In other words, each end of the pipe runs from the moment it receives a value to the moment it sends a value.
The pipe is functionally symmetric, but -- by convention I'm defining in this answer -- the left end is only available inside the generator function's body and is accessible via the yield keyword, while the right end is the generator and is accessible via the generator's send function. As singular interfaces to their respective ends of the pipe, yield and send do double duty: they each both push and pull values to/from their ends of the pipe, yield pushing rightward and pulling leftward while send does the opposite. This double duty is the crux of the confusion surrounding the semantics of statements like x = yield y. Breaking yield and send down into two explicit push/pull steps will make their semantics much more clear:
- Suppose
g is the generator. g.send pushes a value leftward through the right end of the pipe.
- Execution within the context of
g pauses, allowing the generator function's body to run.
- The value pushed by
g.send is pulled leftward by yield and received on the left end of the pipe. In x = yield y, x is assigned to the pulled value.
- Execution continues within the generator function's body until the next line containing
yield is reached.
yield pushes a value rightward through the left end of the pipe, back up to g.send. In x = yield y, y is pushed rightward through the pipe.- Execution within the generator function's body pauses, allowing the outer scope to continue where it left off.
g.send resumes and pulls the value and returns it to the user.- When
g.send is next called, go back to Step 1.
While cyclical, this procedure does have a beginning: when g.send(None) -- which is what next(g) is short for -- is first called (it is illegal to pass something other than None to the first send call). And it may have an end: when there are no more yield statements to be reached in the generator function's body.
Do you see what makes the yield statement (or more accurately, generators) so special? Unlike the measly return keyword, yield is able to pass values to its caller and receive values from its caller all without terminating the function it lives in! (Of course, if you do wish to terminate a function -- or a generator -- it's handy to have the return keyword as well.) When a yield statement is encountered, the generator function merely pauses, and then picks back up right where it left off upon being sent another value. And send is just the interface for communicating with the inside of a generator function from outside it.
If we really want to break this push/pull/pipe analogy down as far as we can, we end up with the following pseudocode that really drives home that, aside from steps 1-5, yield and send are two sides of the same coin pipe:
right_end.push(None) # the first half of g.send; sending None is what starts a generatorright_end.pause()left_end.start()initial_value = left_end.pull()if initial_value is not None: raise TypeError("can't send non-None value to a just-started generator")left_end.do_stuff()left_end.push(y) # the first half of yieldleft_end.pause()right_end.resume()value1 = right_end.pull() # the second half of g.sendright_end.do_stuff()right_end.push(value2) # the first half of g.send (again, but with a different value)right_end.pause()left_end.resume()x = left_end.pull() # the second half of yieldgoto 6
The key transformation is that we have split x = yield y and value1 = g.send(value2) each into two statements: left_end.push(y) and x = left_end.pull(); and value1 = right_end.pull() and right_end.push(value2). There are two special cases of the yield keyword: x = yield and yield y. These are syntactic sugar, respectively, for x = yield None and _ = yield y # discarding value.
For specific details regarding the precise order in which values are sent through the pipe, see below.
What follows is a rather long concrete model of the above. First, it should first be noted that for any generator g, next(g) is exactly equivalent to g.send(None). With this in mind we can focus only on how send works and talk only about advancing the generator with send.
Suppose we have
def f(y): # This is the "generator function" referenced above
while True:
x = yield y
y = x
g = f(1)
g.send(None) # yields 1
g.send(2) # yields 2
Now, the definition of f roughly desugars to the following ordinary (non-generator) function:
def f(y):
bidirectional_pipe = BidirectionalPipe()
left_end = bidirectional_pipe.left_end
right_end = bidirectional_pipe.right_end
def impl():
initial_value = left_end.pull()
if initial_value is not None:
raise TypeError(
"can't send non-None value to a just-started generator"
)
while True:
left_end.push(y)
x = left_end.pull()
y = x
def send(value):
right_end.push(value)
return right_end.pull()
right_end.send = send
# This isn't real Python; normally, returning exits the function. But
# pretend that it's possible to return a value from a function and then
# continue execution -- this is exactly the problem that generators were
# designed to solve!
return right_end
impl()
The following has happened in this transformation of f:
- We've moved the implementation into a nested function.
- We've created a bidirectional pipe whose
left_end will be accessed by the nested function and whose right_end will be returned and accessed by the outer scope -- right_end is what we know as the generator object.
- Within the nested function, the very first thing we do is check that
left_end.pull() is None, consuming a pushed value in the process.
- Within the nested function, the statement
x = yield y has been replaced by two lines: left_end.push(y) and x = left_end.pull().
- We've defined the
send function for right_end, which is the counterpart to the two lines we replaced the x = yield y statement with in the previous step.
In this fantasy world where functions can continue after returning, g is assigned right_end and then impl() is called. So in our example above, were we to follow execution line by line, what would happen is roughly the following:
left_end = bidirectional_pipe.left_end
right_end = bidirectional_pipe.right_end
y = 1 # from g = f(1)
# None pushed by first half of g.send(None)
right_end.push(None)
# The above push blocks, so the outer scope halts and lets `f` run until
# *it* blocks
# Receive the pushed value, None
initial_value = left_end.pull()
if initial_value is not None: # ok, `g` sent None
raise TypeError(
"can't send non-None value to a just-started generator"
)
left_end.push(y)
# The above line blocks, so `f` pauses and g.send picks up where it left off
# y, aka 1, is pulled by right_end and returned by `g.send(None)`
right_end.pull()
# Rinse and repeat
# 2 pushed by first half of g.send(2)
right_end.push(2)
# Once again the above blocks, so g.send (the outer scope) halts and `f` resumes
# Receive the pushed value, 2
x = left_end.pull()
y = x # y == x == 2
left_end.push(y)
# The above line blocks, so `f` pauses and g.send(2) picks up where it left off
# y, aka 2, is pulled by right_end and returned to the outer scope
right_end.pull()
x = left_end.pull()
# blocks until the next call to g.send
This maps exactly to the 16-step pseudocode above.
There are some other details, like how errors are propagated and what happens when you reach the end of the generator (the pipe is closed), but this should make clear how the basic control flow works when send is used.
Using these same desugaring rules, let's look at two special cases:
def f1(x):
while True:
x = yield x
def f2(): # No parameter
while True:
x = yield x
For the most part they desugar the same way as f, the only differences are how the yield statements are transformed:
def f1(x):
# ... set up pipe
def impl():
# ... check that initial sent value is None
while True:
left_end.push(x)
x = left_end.pull()
# ... set up right_end
def f2():
# ... set up pipe
def impl():
# ... check that initial sent value is None
while True:
left_end.push(x)
x = left_end.pull()
# ... set up right_end
In the first, the value passed to f1 is pushed (yielded) initially, and then all values pulled (sent) are pushed (yielded) right back. In the second, x has no value (yet) when it first come times to push, so an UnboundLocalError is raised.
send()is called to start the generator, it must be called withNoneas the argument, because there is no yield expression that could receive the value.", quoted from the official doc and for which the citation in the question is missing. – Rick