Argument binding of python 3 lambda appears to be broken

0
votes

I'm using Python 3.6.10 in a CentOS 7 environment. I am attempting to create a list of commands to execute based on a structured specification. It seems natural and pythonic to think of this as a list of lambdas. I build the lambda list by traversing the specification. To my surprise, when I execute the result, I find that every lambda is the same because it doesn't capture its argument at the time the lambda is created. I think that's a bug.

Here is sample code that illustrates the behavior:

specification = {
    'labelOne': ['labelOne.one', 'labelOne.two', 'labelOne.three', 'labelOne.four', 'labelOne.five'],
    'labelTwo': ['labelTwo.one', 'labelTwo.two', 'labelTwo.three', 'labelTwo.four', 'labelTwo.five'],
    'labelThree': ['labelThree.one', 'labelThree.two', 'labelThree.three', 'labelThree.four', 'labelThree.five'],
    'labelFour': ['labelFour.one', 'labelFour.two', 'labelFour.three', 'labelFour.four', 'labelFour.five'],
    'labelFive': ['labelFive.one', 'labelFive.two', 'labelFive.three', 'labelFive.four', 'labelFive.five'],
    }

lambdas = []
for label, labelStrings in specification.items():
    for labelString in labelStrings:
        lambdaString = f"""Label: \"{label}\" with labelString: \"{labelString}\""""
        oneArgLambda = lambda someArg: print(someArg, lambdaString)
        lambdas.append(oneArgLambda)

for each in lambdas:
    each('Show: ')

I expected to see this:

Show:  Label: "labelOne" with labelString: "labelOne.one"
Show:  Label: "labelOne" with labelString: "labelOne.two"
Show:  Label: "labelOne" with labelString: "labelOne.three"
Show:  Label: "labelOne" with labelString: "labelOne.four"
Show:  Label: "labelOne" with labelString: "labelOne.five"
Show:  Label: "labelTwo" with labelString: "labelTwo.one"
Show:  Label: "labelTwo" with labelString: "labelTwo.two"
Show:  Label: "labelTwo" with labelString: "labelTwo.three"
Show:  Label: "labelTwo" with labelString: "labelTwo.four"
Show:  Label: "labelTwo" with labelString: "labelTwo.five"
Show:  Label: "labelThree" with labelString: "labelThree.one"
Show:  Label: "labelThree" with labelString: "labelThree.two"
Show:  Label: "labelThree" with labelString: "labelThree.three"
Show:  Label: "labelThree" with labelString: "labelThree.four"
Show:  Label: "labelThree" with labelString: "labelThree.five"
Show:  Label: "labelFour" with labelString: "labelFour.one"
Show:  Label: "labelFour" with labelString: "labelFour.two"
Show:  Label: "labelFour" with labelString: "labelFour.three"
Show:  Label: "labelFour" with labelString: "labelFour.four"
Show:  Label: "labelFour" with labelString: "labelFour.five"
Show:  Label: "labelFive" with labelString: "labelFive.one"
Show:  Label: "labelFive" with labelString: "labelFive.two"
Show:  Label: "labelFive" with labelString: "labelFive.three"
Show:  Label: "labelFive" with labelString: "labelFive.four"
Show:  Label: "labelFive" with labelString: "labelFive.five"

Instead, I see this:

Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"
Show:  Label: "labelFive" with labelString: "labelFive.five"

The argument binding of lambda is happening when the lambda is executed, rather than when the lambda is created. That is at least unexpected and I think arguably wrong.

I think lambda, as limited as it is, is supposed to create a CLOSURE -- its entire purpose in life is to capture the state its arguments at the time it is created, so that they can be used later when the lambda is evaluated. That's why it is called a "closure", because it closes over the value of its arguments at creation time.

What am I misunderstanding?

3
pythonclosures
Python lambdas are late binding. Try this instead lambda someArg, ls=lambdaString: print(someArg, ls) - TheLazyScripter
as @TheLazyScripter mentioned, the value is of lambdaString is not evaluated at the time creating oneArgLambda and it takes the latest value when you evaluate it in the loop - Reza
This isn't a lambda-specific issue. Some relevant discussion: docs.python-guide.org/writing/gotchas/#late-binding-closures - FMc

3 Answers

0
votes

This is an alternative

lambdas = []
for label, labelStrings in specification.items():
    for labelString in labelStrings:
        lambdaString = f"""Label: \"{label}\" with labelString: \"{labelString}\""""
        oneArgLambda = lambda someArg, lambdaString: print(someArg, lambdaString)
        lambdas.append((oneArgLambda, lambdaString))

for f, lambdaString in lambdas:
    f('Show: ', lambdaString)
0
votes

As you said it create CLOSURE, and closure is using stated variable in upper scope of lambdaString, but the trick is that all of your lambdas uses the same ref to lambdaString and since you change it every time it remembers the last one. For instance:

c = ['one', 'two']
res = []

for i in range(2):
    for y in c:
        def la(x):
            print(x, y)
        res.append(la)

for la in res:
    la('Show: ')
# Show:  two
# Show:  two
# Show:  two
# Show:  two

You just need another closure to prevent this

c = ['one', 'two']
res = []

for i in range(2):
    for y in c:
        def closure_y(y):
            def la(x):
                print(x, y)
            return la
            
        res.append(closure_y(y))

for la in res:
    la('Show: ')
# Show:  one
# Show:  two
# Show:  one 
# Show:  two

Full code might be

specification = {
    'labelOne': ['labelOne.one', 'labelOne.two', 'labelOne.three', 'labelOne.four', 'labelOne.five'],
    'labelTwo': ['labelTwo.one', 'labelTwo.two', 'labelTwo.three', 'labelTwo.four', 'labelTwo.five'],
    'labelThree': ['labelThree.one', 'labelThree.two', 'labelThree.three', 'labelThree.four', 'labelThree.five'],
    'labelFour': ['labelFour.one', 'labelFour.two', 'labelFour.three', 'labelFour.four', 'labelFour.five'],
    'labelFive': ['labelFive.one', 'labelFive.two', 'labelFive.three', 'labelFive.four', 'labelFive.five'],
    }

lambdas = []
for label, labelStrings in specification.items():
    for labelString in labelStrings:
        lambdaString = f"""Label: \"{label}\" with labelString: \"{labelString}\""""

        def clousure(lambdaString):
            oneArgLambda = lambda someArg: print(someArg, lambdaString)
            return oneArgLambda

        lambdas.append(clousure(lambdaString))

for each in lambdas:
    each('Show: ')
0
votes

You have some other comments and answers explaining what's going on, and your question is "interesting" is the sense that it forces the reader to puzzle over the code and the various binding issues.

But if I saw your code from a colleague during a review, I would ask for a rewrite – not because I would immediately know there was a bug, but because it requires too much head-scratching over whether the bindings from the surrounding (and changing) scope will behave exactly as hoped.

Instead, insist on stricter discipline in your programs, and thus lighten the cognitive load on your reader (who is you most of the time). Specifically, move the function creation to a truly isolated scope, and pass all of the varying inputs to that function-creator. That approach is solid because it will either work on the first try or fail entirely (if you neglect to pass all of the needed arguments to the function-creator).

One way to do it:

# A function to create another function, with non-surprising argument binding.
# We expect nothing from the surrounding scope. All business can be done locally.
def get_func(label, x):
    return lambda prefix: print(f'''{prefix} => {label}: {x}''')

# Some input data.
specification = {
    label : [label + str(n) for n in range(3)]
    for label in ('A', 'B', 'C')
}

# Use that data to create some functions.
funcs = [
    get_func(label, x)
    for label, xs in specification.items()
    for x in xs
]

# Run 'em.
for f in funcs:
    f('Show')