Testing with pytest always issues a file not found error

[vladipus]

Whenever I run the tests, I always getting this strange error:
Detail: Unable to read file (Error: File not found

I really don't have a c file in my project. But where is it coming from?

[vladipus]

This seems to happen due to inline >>> tests, actually.

[luabud]

I'm getting a similar error:

Environment data

• VS Code version: 1.36.1
• Extension version (available under the Extensions sidebar): 2019.6.24221
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.3
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python packages and their versions: pytest version 5.0.1

Expected behaviour

no "file not found" errors being displayed in the output channel when running tests

Actual behaviour

"file not found" error is displayed in the output channel. In the case below, for some reason the "tests" folder is being ignored in the file path:

Steps to reproduce:

1. Open a folder with tests following this structure:
   

2. Have at least one of the tests methods fail

3. Make sure you have a pytest.ini file in the tests folder (in this case I only had [pytest] addopts = -s -v in it)

4. Reload the window and run all tests. Watch the following error appear:
   

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

python C:\Users\luabud\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear tests
============================= test session starts =============================
platform win32 -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- C:\Users\luabud\AppData\Local\Programs\Python\Python37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('c:\\Users\\luabud\\Apps\\vscode\\.hypothesis\\examples')
rootdir: c:\Users\luabud\Apps\vscode\tests, inifile: pytest.ini
plugins: hypothesis-4.24.4
collecting ... collected 8 items

tests\one\test_a.py::test_a_1 PASSED
tests\one\test_a.py::test_a_2 PASSED
tests\one\test_b.py::test_b_1 PASSED
tests\one\test_b.py::test_b_2 PASSED
tests\two\test_c.py::test_c_1 PASSED
tests\two\test_c.py::test_c_2 PASSED
tests\two\test_d.py::test_d_1 PASSED
tests\two\test_d.py::test_d_2 FAILED

================================== FAILURES ===================================
__________________________________ test_d_2 ___________________________________

    def test_d_2():
>       assert 0 == 1
E       assert 0 == 1
E         -0
E         +1

tests\two\test_d.py:6: AssertionError
- generated xml file: C:\Users\luabud\AppData\Local\Temp\tmp-15700c0uP9mjmCb59.xml -
===================== 1 failed, 7 passed in 0.08 seconds ======================
Error: Error: cannot open file:///c%3A/Users/luabud/Apps/vscode/two/test_d.py. Detail: Unable to read file (Error: File not found (c:\Users\luabud\Apps\vscode\two\test_d.py))

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

XXX

[karrtikr]

This seems to happen due to inline >>> tests, actually.

Can you please paste the directory structure and content, like @luabud did. Will help to reproduce the issue.

[karrtikr]

@luabud I can confirm that if test directory contains an pytest.ini, and file path for the files in the test directory are being incorrectly calculated while running tests.

For eg.
Expected path: c:\Users\luabud\Apps\vscode\tests\two\test_d.py
Actual path: c:\Users\luabud\Apps\vscode\two\test_d.py

[vistatest]

Not sure if it the same bug or not. Getting File not found on Mac OS for pytest when click on shaded Run Test above the test. The file and the test obviously exist. Error is below.

============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/vistatest/Git/vista
plugins: postgresql-1.4.0, flask-0.14.0, mock-1.10.4, flask-sqlalchemy-1.0.2
collected 0 items

- generated xml file: /var/folders/lv/_hsxgz4926343l7tn7tc43wm0000gn/T/tmp-1213osfyIlUdGERC.xml -
========================= no tests ran in 0.01 seconds =========================
ERROR: file not found: ./git_work/test_fsrepo.py::test_fsrepo_local_stats

[siben168]

i have the exactly same issue as following when the file is actually in the folder, "d:\workspace\nebula_data\folderabc\tests\test_china_stock_basics.py"

Error Detail: Unable to read file (Error: File not found (d:\workspace\nebula_data\tests\test_china_stock_basics.py))

seems the pytest incorrectly calculated the path of file by ignoring its parently folder, but it wired that actually other test files in the same folder works well.

[LightCC]

I'm seeing the same issue with the most basic of tests. Running from the terminal from within VS Code works fine. Using the test runner appears to work okay from the test run (it says the 1 test passed), but generates an error that appears to keep the output from been sent to the test summary window.

Test and class file are just directly in a folder that I have opened as the workspace.

Output from test output window:

python C:\Users\210068178\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear .
============================= test session starts =============================
platform win32 -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
rootdir: C:\Code\python\ps_unit_testing_with_python
collected 1 item

test_phonebook.py .                                                      [100%]

- generated xml file: C:\Users\210068~1\AppData\Local\Temp\tmp-25948AQtNhyZ0M6Ak.xml -
============================== 1 passed in 0.05s ==============================
Error: TypeError: Cannot read property '$' of undefined

Output when running from terminal:

$ py -m pytest
======================================== test session starts =========================================
platform win32 -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
rootdir: C:\Code\python\ps_unit_testing_with_python
collected 1 item                                                                                      

test_phonebook.py .                                                                             [100%]

========================================= 1 passed in 0.04s ==========================================
(.venv)
210068178@GCXLSHV2E MINGW64 /c/Code/python/ps_unit_testing_with_python (module_3_basic-examples-with-pytest)
$

VS Code Help Info:

Version: 1.37.1 (user setup)
Commit: f06011ac164ae4dc8e753a3fe7f9549844d15e35
Date: 2019-08-15T16:17:55.855Z
Electron: 4.2.7
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Windows_NT x64 10.0.17763

[luabud]

@LightCC thanks for reporting this ! The issue you're seeing seems to be this one: #6990, and it's related to pytest 5.1. We're working hard to get it fixed.

[elusive]

fwiw, I was seeing the issue #6990 but then i downgraded to pytest 5.01 and that issue went away. but i still have this issue (file not found on all tests i execute).

[luc99hen]

I have met similar problem with @luabud . And the werid thing is that it works fine when I start tests by clicking the button from the side panel.

and fails for all the other situation.

[CMLL]

I've also ran into the above issue.

Log output:

-------------- generated xml file: /tmp/tmp-25268acIAnO16YeGR.xml --------------
========================= no tests ran in 0.00 seconds =========================
ERROR: file not found: ./unit/dashboards/test_widget_timescale.py

============================= test session starts ==============================
platform linux2 -- Python 2.7.15+, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/cllamach/Panopta/classic
plugins: xdist-1.28.0, sugar-0.9.2, pythonpath-0.7.3, forked-1.0.2, env-0.6.2, pylama-7.7.1

-------------- generated xml file: /tmp/tmp-25268YJtfJRChbkNv.xml --------------
========================= no tests ran in 0.00 seconds =========================
ERROR: file not found: ./unit/dashboards/test_widget_timescale.py::test_parse_returns_timescale_at_right_timezeone

File Structure:

src/
    tests/
            unit/
                  dashboards/
                          test_widget_timescale.py

Settings:

    "python.testing.pytestArgs": [
        "src/tests/"
    ],

[ericsnowcurrently]

As has already been noted, if there is a pytest.ini file in the test root then we run into this problem. I was able to verify that this is not a problem if the file is in any other directory, even a sub-directory of the test root.

I also found that this bug is impacted by test discovery. The following two cases indicate that test execution is tied to test discovery:

• remove pytest.ini after discovery
  1. add pytest.ini in the test root
  2. discover tests
  3. remove pytest.ini
  4. run tests (the bug happens unexpectedly)
• add pytest.ini after discovery
  1. discover tests
  2. add pytest.ini in the test root
  3. run tests (the bug does not happen...unexpectedly)

Running with "--cache-clear" did not make a difference.

I also found that running a single test fails because it tries to use a path relative to the test root rather than CWD (the workspace root). This is basically the same problem.

Finally, I was able to reproduce the problem at revision b4aa1f0, which predates the recent refactoring of the XUnit parser (#7263/#7265 for #6990).

[ericsnowcurrently]

The key problem is that, with pytest.ini in the test root, pytest returns filenames (for both discovery and execution) that are relative to the test root. In all other cases they are relative to the workspace root. However, the extension code makes the assumption that the relative filenames are always relative to the workspace root.

In the case of a failed test we do a lookup using the absolute filename (see TestMessageService.getLocationStack()).

This will be resolved if we do one of the following:

1. where used, always resolve the filename relative to the discovered test root
2. during discovery (and execution) store the relative filename correctly resolved to the workspace root (based on the test root)
3. during discovery (and execution) store the absolute filename (merge the given one with the test root, which is always absolute)

I prefer (3).

In the case of running individual tests, that will be resolved if we do one of the following:

1. during discovery store the correct relative filename in "fileToRun" (i.e. relative to workspace root)
2. during discovery store the absolute filename in fileToRun (may not be right)
3. during execution, resolve "fileToRun" against the workspace root (as necessary)

(Note that "fileToRun" is not modified during execution, at least for pytest.)

I prefer (1).

[ericsnowcurrently]

It turns out this is the defined behavior for pytest. It will walk up the directory tree, starting at the given directory/file, and set the first ancestor directory with a pytest.ini file as the root. That is exactly what everyone is seeing. The problem is that the extension isn't coping well with this behavior. It's going to take some cleverness to get this right.

[kimadeline]

✅ Validated using version 2019.11.41015-dev of the extension and this folder structure (all passing tests):

Tests are being discovered and run correctly: