I was wondering why list comprehension is so much faster than appending to a list. I thought the difference is just expressive, but it's not.
>>> import timeit
>>> timeit.timeit(stmt='''\
t = []
for i in range(10000):
t.append(i)''', number=10000)
9.467898777974142
>>> timeit.timeit(stmt='t= [i for i in range(10000)]', number=10000)
4.1138417314859
The list comprehension is 50% faster. Why?
List comprehension is basically just a "syntactic sugar" for the regular for loop. In this case the reason that it performs better is because it doesn't need to load the append attribute of the list and call it as a function at each iteration. In other words and in general, list comprehensions perform faster because suspending and resuming a function's frame, or multiple functions in other cases, is slower than creating a list on demand.
Consider the following examples :
In [1]: def f1():
...: l = []
...: for i in range(5):
...: l.append(i)
...:
...:
...: def f2():
...: [i for i in range(5)]
...:
In [3]: import dis
In [4]: dis.dis(f1)
2 0 BUILD_LIST 0
2 STORE_FAST 0 (l)
3 4 LOAD_GLOBAL 0 (range)
6 LOAD_CONST 1 (5)
8 CALL_FUNCTION 1
10 GET_ITER
>> 12 FOR_ITER 14 (to 28)
14 STORE_FAST 1 (i)
4 16 LOAD_FAST 0 (l)
18 LOAD_METHOD 1 (append)
20 LOAD_FAST 1 (i)
22 CALL_METHOD 1
24 POP_TOP
26 JUMP_ABSOLUTE 12
>> 28 LOAD_CONST 0 (None)
30 RETURN_VALUE
In [5]:
In [5]: dis.dis(f2)
8 0 LOAD_CONST 1 (<code object <listcomp> at 0x7f397abc0d40, file "<ipython-input-1-45c11e415ee9>", line 8>)
2 LOAD_CONST 2 ('f2.<locals>.<listcomp>')
4 MAKE_FUNCTION 0
6 LOAD_GLOBAL 0 (range)
8 LOAD_CONST 3 (5)
10 CALL_FUNCTION 1
12 GET_ITER
14 CALL_FUNCTION 1
16 POP_TOP
18 LOAD_CONST 0 (None)
20 RETURN_VALUE
Disassembly of <code object <listcomp> at 0x7f397abc0d40, file "<ipython-input-1-45c11e415ee9>", line 8>:
8 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 8 (to 14)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LIST_APPEND 2
12 JUMP_ABSOLUTE 4
>> 14 RETURN_VALUE
In [6]:
You can see that on offset 18 in the first function we have an append attribute while there's no such thing in second function using list comprehension. All those extra bytecodes will make the appending approach slower. Also note that you'll also have the append attribute loading in each iteration which makes your code takes approximately 2 time slower than the second function using list comprehension.
Even factoring out the time it takes to lookup and load the append function, the list comprehension is still faster because the list is created in C, rather than built up one item at a time in Python.
# Slow
timeit.timeit(stmt='''
for i in range(10000):
t.append(i)''', setup='t=[]', number=10000)
# Faster
timeit.timeit(stmt='''
for i in range(10000):
l(i)''', setup='t=[]; l=t.append', number=10000)
# Faster still
timeit.timeit(stmt='t = [i for i in range(10000)]', number=10000)
Citing this article, it is because the append attribute of the list isn't looked up, loaded and called as a function, which takes time and that adds up over iterations.
listwhile list comprehension is python's built-in feature. - Iqra.