Why list comprehensions create a function internally?

Question

This is disassembly of a list comprehension in python-3.10:

Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> 
>>> dis.dis("[True for _ in ()]")
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x7fea68e0dc60, file "<dis>", line 1>)
              2 LOAD_CONST               1 ('<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_CONST               2 (())
              8 GET_ITER
             10 CALL_FUNCTION            1
             12 RETURN_VALUE

Disassembly of <code object <listcomp> at 0x7fea68e0dc60, file "<dis>", line 1>:
  1           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)
        >>    4 FOR_ITER                 4 (to 14)
              6 STORE_FAST               1 (_)
              8 LOAD_CONST               0 (True)
             10 LIST_APPEND              2
             12 JUMP_ABSOLUTE            2 (to 4)
        >>   14 RETURN_VALUE

From what I understand it creates a code object called listcomp which does the actual iteration and return the result list, and immediately call it. I can't figure out the need to create a separate function to execute this job. Is this kind of an optimization trick?

Answer 1

The main logic of creating a function is to isolate the comprehension’s iteration variable^{peps.python.org}.

By creating a function:

Comprehension iteration variables remain isolated and don’t overwrite a variable of the same name in the outer scope, nor are they visible after the comprehension

However, this is inefficient at runtime. Due to this reason, python-3.12 implemented an optimization called comprehension inlining(PEP 709)^{peps.python.org} which will no longer create a separate code object^{peps.python.org}.

Dictionary, list, and set comprehensions are now inlined, rather than creating a new single-use function object for each execution of the comprehension. This speeds up execution of a comprehension by up to two times. See PEP 709 for further details.

Here is the output for the same code disassembled with python-3.12:

>>> import dis
>>> 
>>> dis.dis("[True for _ in ()]")
  0           0 RESUME                   0

  1           2 LOAD_CONST               0 (())
              4 GET_ITER
              6 LOAD_FAST_AND_CLEAR      0 (_)
              8 SWAP                     2
             10 BUILD_LIST               0
             12 SWAP                     2
        >>   14 FOR_ITER                 4 (to 26)
             18 STORE_FAST               0 (_)
             20 LOAD_CONST               1 (True)
             22 LIST_APPEND              2
             24 JUMP_BACKWARD            6 (to 14)
        >>   26 END_FOR
             28 SWAP                     2
             30 STORE_FAST               0 (_)
             32 RETURN_VALUE
        >>   34 SWAP                     2
             36 POP_TOP
             38 SWAP                     2
             40 STORE_FAST               0 (_)
             42 RERAISE                  0
ExceptionTable:
  10 to 26 -> 34 [2]

As you can see, there is no longer a MAKE_FUNCTION opcode nor a separate code object. Instead python-3.12 uses LOAD_FAST_AND_CLEAR^{docs.python.org}(at offset 6) and STORE_FAST(at offset 30) opcodes to provide the isolation for the iteration variable.

Quoting from the Specification section^{peps.python.org} of the PEP 709:

Isolation of the x iteration variable is achieved by the combination of the new LOAD_FAST_AND_CLEAR opcode at offset 6, which saves any outer value of x on the stack before running the comprehension, and 30 STORE_FAST, which restores the outer value of x (if any) after running the comprehension.

In addition to that, in python-3.12 there is no longer a separate frame for the comprehension in tracebacks.

Traceback in <python-3.12	Traceback in python-3.12
>>> [1 / 0 for i in range(10)] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <listcomp> ZeroDivisionError: division by zero	>>> [1 / 0 for i in range(10)] Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: division by zero

And here is the benchmark results^{peps.python.org}(measured with MacOS M2):

$ python3.10 -m pyperf timeit -s 'l = [1]' '[x for x in l]'
Mean +- std dev: 108 ns +- 3 ns
$ python3.12 -m pyperf timeit -s 'l = [1]' '[x for x in l]'
Mean +- std dev: 60.9 ns +- 0.3 ns

Answer 2

List comprehensions in Python don't create a function internally. They are a concise way to create lists. If you're referring to lambda functions, they can be used within a list comprehension to apply a function to each element. However, list comprehensions themselves are not functions; they're a syntactic construct for creating lists.

Answer 3

list comprehensions in Python do not create a function internally. List comprehensions are a syntactic construct for creating lists in a concise and readable manner. They are not equivalent to creating or using functions, and they do not involve the creation of functions behind the scenes.

List comprehensions are more closely related to loops, as they provide a compact way to express a loop that constructs a list. They are a shorthand notation for creating lists and are evaluated in a single line. Here's an example to illustrate:

# Using a for loop to create a new list

original_list = [1, 2, 3, 4, 5] new_list = [] for item in original_list: new_list.append(item * 2)

Equivalent list comprehension

new_list_comprehension = [item * 2 for item in original_list]

print(new_list) # Output: [2, 4, 6, 8, 10] print(new_list_comprehension) # Output: [2, 4, 6, 8, 10]

In the example above, the list comprehension [item * 2 for item in original_list] is a concise way to achieve the same result as the for loop without explicitly creating a separate function.

To reiterate, list comprehensions are not functions; they are a specific syntax for creating lists. Functions, on the other hand, are defined using the def keyword and involve a more general mechanism for encapsulating reusable blocks of code.