WIP: #9298 - add testing and debugging for the pycache_prefix

[RonnyPfannschmidt]

adding debugging first, then potentially a xfail for the affected combination (python 3.9 + windows)

[RonnyPfannschmidt]

@The-Compiler accrording to the output, it seems like for some reason the complete set of path parents is replicated

@nicoddemus any idea/suggestion (its potentially a windows flag thats enabled in the CI and not enabled on normal systems)

[nicoddemus]

Here's the output paths, cleaned up:

C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integration0
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integration0/src
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw0/test_sys_pycache_prefix_integration0/src/bar

it seems like for some reason the complete set of path parents is replicated

You mean the full path is being appended to the original path? That's the expected behavior of get_cache_dir:

pytest/src/_pytest/assertion/rewrite.py

Line 1131 in cd49075

return Path(sys.pycache_prefix) / Path(*file_path.parts[1:-1])

The path after /my/pycs is a version of the original path to avoid pyc clashes.

Let me push a commit to this branch to further try to isolate the problem.

[nicoddemus]

Ahh interesting, running that test in isolation still fails on CI.

[nicoddemus]

Interesting, here are the paths in which version:

py38

C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integration0/src/test_foo.cpython-38-pytest-7.0.0.dev322+ga1bc34860.pyc
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integration0/src/bar/__init__.cpython-38.pyc

py39

No files at all.

py310

C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integration0/src/test_foo.cpython-310-pytest-7.0.0.dev322+ga1bc34860.pyc
C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integr0/my/pycs/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_sys_pycache_prefix_integration0/src/bar/__init__.cpython-310.pyc

So the problem is not some file name mismatching, we are not writing .pyc at all in the cache dir.

[nicoddemus]

Ahh OK got to the actual error!

It is happening in _write_pyc:

write_pyc: error! [WinError 3] The system cannot find the path specified.

I suspect long paths.

[nicoddemus]

  write_pyc: parent exists? True
  write_pyc: error! [WinError 3] The system cannot find the path specified.

Yep! The parent folder exists, but we can't write the file. Pretty sure that's to long paths not working there. Will try a quick hack to confirm this.

[nicoddemus]

Yes, the problem is long paths, I changed get_cache_dir to generate smaller paths, and then the pyc file is generated in Python 3.9.

This is related to Python 3.9.8:

actions/setup-python#267

Seems this will solve itself next week with Python 3.9.9.

[nicoddemus]

I suggest we keep the xfail but add strict=True, so it will start to fail when the image is updated, meaning we can then remove that xfail.

[RonnyPfannschmidt]

closing this as we learned it was a python deployment issue