soft_unicode() function: SystemError when DeprecationWarning warnings are not ignored

[vstinner]

The "tests/test_core_tags.py::TestSet::test_block_escaping_filtered" test does fail with a Python built in debug mode if DeprecationWarning warnings are displayed using Python -X dev command line option:
https://docs.python.org/dev/library/devmode.html

The problem seems to come from the soft_unicode() function which returns a result (Markup instance) whereas there is a current exception: DeprecationWarning.

I tested Jinja 2.11.2 with Python 3.10.0a0 on Fedora 32 (x86-64).

Steps to reproduce with Python 3.8 with a regular Python binary:

wget https://files.pythonhosted.org/packages/64/a7/45e11eebf2f15bf987c3bc11d37dcc838d9dc81250e67e4c5968f6008b6c/Jinja2-2.11.2.tar.gz
tar -xf Jinja2-2.11.2.tar.gz 
cd Jinja2-2.11.2/
python3.8 -m venv env
env/bin/python setup.py install
env/bin/python -m pip install pytest
env/bin/python -W default -m pytest -v tests/test_core_tags.py

Result:

(...)
                                                                                                                                 [ 88%]
tests/test_core_tags.py::TestSet::test_namespace_loop PASSED                                                                                                                                                                                               [ 90%]
tests/test_core_tags.py::TestSet::test_namespace_macro PASSED                                                                                                                                                                                              [ 91%]
tests/test_core_tags.py::TestSet::test_block_escaping_filtered FAILED                                                                                                                                                                                      [ 93%]
tests/test_core_tags.py::TestSet::test_block_filtered FAILED                                                                                                                                                                                               [ 95%]
tests/test_core_tags.py::TestSet::test_block_filtered_set FAILED                                                                                                                                                                                           [ 96%]
tests/test_core_tags.py::TestWith::test_with PASSED                                                                                                                                                                                                        [ 98%]
tests/test_core_tags.py::TestWith::test_with_argument_scoping PASSED                                                                                                                                                                                       [100%]

============================================================================================================================ FAILURES ============================================================================================================================
______________________________________________________________________________________________________________ TestSet.test_block_escaping_filtered ______________________________________________________________________________________________________________
DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in version 2.1.

The above exception was the direct cause of the following exception:

self = <test_core_tags.TestSet object at 0x7f9da56ed250>

    def test_block_escaping_filtered(self):
        env = Environment(autoescape=True)
        tmpl = env.from_string(
            "{% set foo | trim %}<em>{{ test }}</em>    {% endset %}foo: {{ foo }}"
        )
>       assert tmpl.render(test="<unsafe>") == "foo: <em>&lt;unsafe&gt;</em>"

tests/test_core_tags.py:553: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:1090: in render
    self.environment.handle_exception()
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:832: in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/_compat.py:28: in reraise
    raise value.with_traceback(tb)
<template>:1: in top-level template code
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = Markup('<em>&lt;unsafe&gt;</em>    '), chars = None

    def do_trim(value, chars=None):
        """Strip leading and trailing characters, by default whitespace."""
>       return soft_unicode(value).strip(chars)
E       SystemError: <built-in function soft_unicode> returned a result with an error set

env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/filters.py:828: SystemError
__________________________________________________________________________________________________________________ TestSet.test_block_filtered ___________________________________________________________________________________________________________________
DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in version 2.1.

The above exception was the direct cause of the following exception:

self = <test_core_tags.TestSet object at 0x7f9da5f974c0>, env_trim = <jinja2.environment.Environment object at 0x7f9da5f975b0>

    def test_block_filtered(self, env_trim):
        tmpl = env_trim.from_string(
            "{% set foo | trim | length | string %} 42    {% endset %}{{ foo }}"
        )
>       assert tmpl.render() == "2"

tests/test_core_tags.py:559: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:1090: in render
    self.environment.handle_exception()
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:832: in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/_compat.py:28: in reraise
    raise value.with_traceback(tb)
<template>:1: in top-level template code
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = ' 42    ', chars = None

    def do_trim(value, chars=None):
        """Strip leading and trailing characters, by default whitespace."""
>       return soft_unicode(value).strip(chars)
E       SystemError: <built-in function soft_unicode> returned a result with an error set

env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/filters.py:828: SystemError
________________________________________________________________________________________________________________ TestSet.test_block_filtered_set _________________________________________________________________________________________________________________
DeprecationWarning: 'soft_unicode' has been renamed to 'soft_str'. The old name will be removed in version 2.1.

The above exception was the direct cause of the following exception:

self = <test_core_tags.TestSet object at 0x7f9da56e3790>, env_trim = <jinja2.environment.Environment object at 0x7f9da56e3be0>

    def test_block_filtered_set(self, env_trim):
        def _myfilter(val, arg):
            assert arg == " xxx "
            return val
    
        env_trim.filters["myfilter"] = _myfilter
        tmpl = env_trim.from_string(
            '{% set a = " xxx " %}'
            "{% set foo | myfilter(a) | trim | length | string %}"
            ' {% set b = " yy " %} 42 {{ a }}{{ b }}   '
            "{% endset %}"
            "{{ foo }}"
        )
>       assert tmpl.render() == "11"

tests/test_core_tags.py:575: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:1090: in render
    self.environment.handle_exception()
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/environment.py:832: in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/_compat.py:28: in reraise
    raise value.with_traceback(tb)
<template>:1: in top-level template code
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = '  42  xxx  yy    ', chars = None

    def do_trim(value, chars=None):
        """Strip leading and trailing characters, by default whitespace."""
>       return soft_unicode(value).strip(chars)
E       SystemError: <built-in function soft_unicode> returned a result with an error set

env/lib64/python3.8/site-packages/Jinja2-2.11.2-py3.8.egg/jinja2/filters.py:828: SystemError

Python traceback:

(gdb) p _Py_DumpTraceback(3, tstate)
Stack (most recent call first):
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/Jinja2-2.11.2-py3.10.egg/jinja2/filters.py", line 828 in do_trim
  File "<template>", line 21 in root
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/Jinja2-2.11.2-py3.10.egg/jinja2/environment.py", line 1088 in render
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/Jinja2-2.11.2-py3.10.egg/jinja2/asyncsupport.py", line 71 in render
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/Jinja2-2.11.2/tests/test_core_tags.py", line 553 in test_block_escaping_filtered
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/_pytest/python.py", line 170 in pytest_pyfunc_call
  File "/home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
(...)

Raw debug logs:

vstinner@apu$ gdb -args /home/vstinner/myprojects/pythonci/work/cpython-3.10.0a0_Jinja2-2.11.2/venv/bin/python -X dev -m pytest --tb=short -v 
(...)
(gdb) b _Py_FatalErrorFunc 
Breakpoint 1 at 0x56ca7c: file Python/pylifecycle.c, line 2280.
(gdb) b fatal_error
Breakpoint 2 at 0x56c894: file Python/pylifecycle.c, line 2192.

(gdb) run
(...)
tests/test_core_tags.py::TestSet::test_block_escaping_filtered 
Breakpoint 1, _Py_FatalErrorFunc (func=0x70b250 <__func__.19> "_Py_CheckFunctionResult", msg=0x70aec8 "a function returned a result with an error set") at Python/pylifecycle.c:2280
2280	    fatal_error(stderr, 1, func, msg, -1);
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.8-2.fc32.x86_64 expat-2.2.8-2.fc32.x86_64 libuuid-2.35.2-1.fc32.x86_64 libxcrypt-4.4.16-3.fc32.x86_64 openssl-libs-1.1.1g-1.fc32.x86_64 xz-libs-5.2.5-1.fc32.x86_64 zlib-1.2.11-21.fc32.x86_64
(gdb) up
#1  0x00000000004301f0 in _Py_CheckFunctionResult (tstate=0x89a780, callable=0x7fffe9d63b90, result=0x7fffe8b11270, where=0x0) at Objects/call.c:61
61	            Py_FatalError("a function returned a result with an error set");
(gdb) p tstate
$1 = (PyThreadState *) 0x89a780
(gdb) p tstate->curexc_type 
$2 = (PyObject *) 0x846c20 <_PyExc_DeprecationWarning>

[davidism]

Thanks, looks like the expected result when turning deprecation warnings into errors. It seems that you're using a pre-release of MarkupSafe, 2.0.0a1; the current release does not yet raise this warning. We should of course update the code here to use the preferred name going forward.

[vstinner]

It seems that you're use a pre-release of MarkupSafe, 2.0.0a1

Right. But I didn't pick this version on purpose, it's what I get by default when running python setup.py install. setup.py contains:

install_requires=["MarkupSafe>=0.23"],

If you follow my wget, tar, etc. instructions, you get a venv with these dependencies:

$ env/bin/python -m pip freeze
attrs==19.3.0
iniconfig==1.0.1
Jinja2==2.11.2
MarkupSafe==2.0.0a1
more-itertools==8.4.0
packaging==20.4
pluggy==0.13.1
py==1.9.0
pyparsing==2.4.7
pytest==6.0.1
six==1.15.0
toml==0.10.1

When I install MarkupSafe manually with pip, I get MarkupSafe 1.1.1.

$ python3.8 -m venv env
$ env/bin/python -m pip install MarkupSafe
(...)
Successfully installed MarkupSafe-1.1.1

$ env/bin/python -m pip freeze
MarkupSafe==1.1.1

[davidism]

setup.py install is deprecated, it does not support most modern features expected in packaging, such as python_requires or not picking pre-releases by default. Use pip install to install packages.