python - What do (lambda) function closures capture?

Question

Recently I started playing around with Python and I came around something peculiar in the way closures work. Consider the following code:

adders=[None, None, None, None]

for i in [0,1,2,3]:
   adders[i]=lambda a: i+a

print adders[1](3)

It builds a simple array of functions that take a single input and return that input added by a number. The functions are constructed in for loop where the iterator i runs from 0 to 3. For each of these numbers a lambda function is created which captures i and adds it to the function's input. The last line calls the second lambda function with 3 as a parameter. To my surprise the output was 6.

I expected a 4. My reasoning was: in Python everything is an object and thus every variable is essential a pointer to it. When creating the lambda closures for i, I expected it to store a pointer to the integer object currently pointed to by i. That means that when i assigned a new integer object it shouldn't effect the previously created closures. Sadly, inspecting the adders array within a debugger shows that it does. All lambda functions refer to the last value of i, 3, which results in adders[1](3) returning 6.

Which make me wonder about the following:

• What do the closures capture exactly?
• What is the most elegant way to convince the lambda functions to capture the current value of i in a way that will not be affected when i changes its value?

Answer 1

Your second question has been answered, but as for your first:

what does the closure capture exactly?

Scoping in Python is dynamic and lexical. A closure will always remember the name and scope of the variable, not the object it's pointing to. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

EDIT: Regarding your other question of how to overcome this, there are two ways that come to mind:

1. The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument's default value to the object you want preserved.

2. A little more verbose but less hacky would be to create a new scope each time you create the lambda:

   >>> adders = [0,1,2,3]
   >>> for i in [0,1,2,3]:
   ...     adders[i] = (lambda b: lambda a: b + a)(i)
   ...     
   >>> adders[1](3)
   4
   >>> adders[2](3)
   5
   

   The scope here is created using a new function (a lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope:

   def createAdder(x):
       return lambda y: y + x
   adders = [createAdder(i) for i in range(4)]
   

Answer 2

you may force the capture of a variable using an argument with a default value:

>>> for i in [0,1,2,3]:
...    adders[i]=lambda a,i=i: i+a  # note the dummy parameter with a default value
...
>>> print( adders[1](3) )
4

the idea is to declare a parameter (cleverly named i) and give it a default value of the variable you want to capture (the value of i)

Answer 3

For completeness another answer to your second question: You could use partial in the functools module.

With importing add from operator as Chris Lutz proposed the example becomes:

from functools import partial
from operator import add   # add(a, b) -- Same as a + b.

adders = [0,1,2,3]
for i in [0,1,2,3]:
   # store callable object with first argument given as (current) i
   adders[i] = partial(add, i) 

print adders[1](3)

Answer 4

Consider the following code:

x = "foo"

def print_x():
    print x

x = "bar"

print_x() # Outputs "bar"

I think most people won't find this confusing at all. It is the expected behaviour.

So, why do people think it would be different when it is done in a loop? I know I did that mistake myself, but I don't know why. It is the loop? Or perhaps the lambda?

After all, the loop is just a shorter version of:

adders= [0,1,2,3]
i = 0
adders[i] = lambda a: i+a
i = 1
adders[i] = lambda a: i+a
i = 2
adders[i] = lambda a: i+a
i = 3
adders[i] = lambda a: i+a

Answer 5

Here's a new example that highlights the data structure and contents of a closure, to help clarify when the enclosing context is "saved."

def make_funcs():
    i = 42
    my_str = "hi"

    f_one = lambda: i

    i += 1
    f_two = lambda: i+1

    f_three = lambda: my_str
    return f_one, f_two, f_three

f_1, f_2, f_3 = make_funcs()

What is in a closure?

>>> print f_1.func_closure, f_1.func_closure[0].cell_contents
(<cell at 0x106a99a28: int object at 0x7fbb20c11170>,) 43 

Notably, my_str is not in f1's closure.

What's in f2's closure?

>>> print f_2.func_closure, f_2.func_closure[0].cell_contents
(<cell at 0x106a99a28: int object at 0x7fbb20c11170>,) 43

Notice (from the memory addresses) that both closures contain the same objects. So, you can start to think of the lambda function as having a reference to the scope. However, my_str is not in the closure for f_1 or f_2, and i is not in the closure for f_3 (not shown), which suggests the closure objects themselves are distinct objects.

Are the closure objects themselves the same object?

>>> print f_1.func_closure is f_2.func_closure
False

Answer 6

In answer to your second question, the most elegant way to do this would be to use a function that takes two parameters instead of an array:

add = lambda a, b: a + b
add(1, 3)

However, using lambda here is a bit silly. Python gives us the operator module, which provides a functional interface to the basic operators. The lambda above has unnecessary overhead just to call the addition operator:

from operator import add
add(1, 3)

I understand that you're playing around, trying to explore the language, but I can't imagine a situation I would use an array of functions where Python's scoping weirdness would get in the way.

If you wanted, you could write a small class that uses your array-indexing syntax:

class Adders(object):
    def __getitem__(self, item):
        return lambda a: a + item

adders = Adders()
adders[1](3)

Answer 7

One way to sort out the scope of i is to generate the lambda in another scope (a closure function), handing over the necessary parameters for it to make the lambda:

def get_funky(i):
    return lambda a: i+a

adders=[None, None, None, None]

for i in [0,1,2,3]:
   adders[i]=get_funky(i)

print(*(ar(5) for ar in adders))

giving 5 6 7 8 of course.