Jumping before comprehensions yields SIGSEGV

[dkrystki]

Full reproducer:

import sys


def fun():
    a = 1  # <---- jump location

    x = [i for i in range(10)]

    c = 3  # current location


def trace(frame, event, arg):
    if event == "line" and frame.f_lineno == 9:
        frame.f_lineno = 4

    return trace


sys.settrace(trace)

if __name__ == "__main__":
    fun()

Jumping from line current location to jump location yields SIGSEGV

Error messages

.venv/bin/python <cwd>/cakeshop.py
Fatal Python error: Segmentation fault

Current thread 0x00007f16eaae5280 (most recent call first):
  File "<cwd>/cakeshop.py", line 14 in trace
  File "<cwd>/cakeshop.py", line 9 in fun
  File "<cwd>/cakeshop.py", line 22 in <module>

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Your environment

• CPython versions tested on: 3.9.11, 3.10.1 (3.11.x works as expected same as <3.9)
• Operating system and architecture: Linux q 5.4.0-42-generic x64

[serhiy-storchaka]

@markshannon

[sweeneyde]

Bisecting between v3.10.0a1 and main, it looks like this was the first commit in which the behavior was fixed:

adcd2205565f91c6719f4141ab4e1da6d7086126 is the first new commit
commit adcd2205565f91c6719f4141ab4e1da6d7086126
Author: Mark Shannon <mark@hotpy.org>
Date:   Fri May 7 15:19:19 2021 +0100

    bpo-40222: "Zero cost" exception handling (GH-25729)

    "Zero cost" exception handling.

    * Uses a lookup table to determine how to handle exceptions.
    * Removes SETUP_FINALLY and POP_TOP block instructions, eliminating (most of) the runtime overhead of try statements.
    * Reduces the size of the frame object by about 60%.

Bisecting between v3.8.0a1 and 3.10.0a1 got me this as the breaking commit:

fee552669f21ca294f57fe0df826945edc779090 is the first bad commit
commit fee552669f21ca294f57fe0df826945edc779090
Author: Mark Shannon <mark@hotpy.org>
Date:   Thu Nov 21 09:11:43 2019 +0000

    Produce cleaner bytecode for 'with' and 'async with' by generating separate code for normal and exceptional paths. (#6641)

    Remove BEGIN_FINALLY, END_FINALLY, CALL_FINALLY and POP_FINALLY bytecodes. Implement finally blocks by code duplication.
    Reimplement frame.lineno setter using line numbers rather than bytecode offsets.

[sweeneyde]

This is a walkthrough of the offending call to frame_setlineno in Python v3.10.0a1:

The relevant bytecode of fun()

  5           0 LOAD_CONST               1 (1)
              2 STORE_FAST               0 (a)

  7           4 LOAD_CONST               2 (<code object <listcomp> ...>)
              6 LOAD_CONST               3 ('fun.<locals>.<listcomp>')
              8 MAKE_FUNCTION            0
             10 LOAD_GLOBAL              0 (range)
             12 LOAD_CONST               4 (10)
             14 CALL_FUNCTION            1
             16 GET_ITER
             18 CALL_FUNCTION            1
             20 STORE_FAST               1 (x)

  9          22 LOAD_CONST               5 (3)
             24 STORE_FAST               2 (c)
             26 LOAD_CONST               0 (None)
             28 RETURN_VALUE

Walking through frame_setlineno

new_lineno is 4.
f_lasti is 22 (byte 22, codeunit 11).

marklines produces: int *lines = {
     [0] = 5,   [1] = -1, [2] = 7,   [3] = -1,  [4] = -1, 
     [5] = -1,  [6] = -1, [7] = -1,  [8] = -1,  [9] = -1,
    [10] = -1, [11] = 9, [12] = -1, [13] = -1, [14] = -1,
};

first_line_not_before sets new_lineno becomes 5.

markblocks produces: int64_t *blocks = {
     [0] = 0,  [1] = 0,  [2] = 0,  [3] = 0,  [4] = 0,
     [5] = 0,  [6] = 0,  [7] = 0,  [8] = 0,  [9] = 2,
    [10] = 2, [11] = 2, [12] = 2, [13] = 2, [14] = 2,
};

start_block_stack is 2.

lines[0] == new_lineno.
target_blocks_stack is 0.

compatible_block_stack(from_stack, to_stack),
    from_stack is 2,
    to_stack is 0.
    return true

best_block_stack becomes 0.
best_attr becomes 0.
top_block(start_block_stack) -> top_block(2) -> Loop (2)
fram_stack_pop() is called.

f->f_stackdepth is 0
f->f_stackdepth-- to -1.
PyObject *v = f->f_valuestack[f->f_stackdepth];
v is NULL.
Py_DECREF(NULL);
segmentation fault.

So I believe the problem is markblocks(): I don't know why it would decide that a Loop (2) is on the stack at indices 9-14.

[sweeneyde]

Ah, so this is the issue in 3.10:

cpython/Objects/frameobject.c

Lines 183 to 187 in 8e9afaf

	case GET_ITER:
	case GET_AITER:
	block_stack = push_block(block_stack, Loop);
	blocks[i+1] = block_stack;
	break;

The code assumed that GET_ITER must signal the start of a for-loop, so it pushes a Loop (2) code. But it's not a for-loop; it's a list comprehension. So the Loop block is never popped.

[sweeneyde]

• Added tests to main branch: gh-92311: Add tests for frame_setlineno jumping over listcomps #92741
• Adding tests to 3.11: [3.11] gh-92311: Add tests for frame_setlineno jumping over listcomps (GH-92741) #92797
• Fixed 3.10: [3.10] gh-92311: Let frame_setlineno jump over listcomps #92717
• Fixed 3.9: [3.9] gh-92311: Let frame_setlineno jump over listcomps #92740