Error in Test Debug Mode: Cannot read properties of null (testsuites)

[pamelafox]

Type: Bug

Behaviour

Expected vs. Actual

I expect debug mode to work for the testing tab, but it fails with an error.

Steps to reproduce:

1. Check out this branch:
   Sept 18 update from cruft Azure-Samples/azure-flask-postgres-flexible-appservice#1
2. Open in a Dev Container
3. Install Playwright per the README instructions
4. Run tests from testing tab, they should pass.
5. Run test with debug button from testing tab, see error:

Running tests (pytest): /workspace
Running test with arguments: --rootdir /workspace --override-ini junit_family=xunit1 --junit-xml=/tmp/tmp-3118LSNEbwR49z8.xml --capture no
Current working directory: /workspace
Workspace directory: /workspace
Run completed, parsing output
Error while running tests: workspace
TypeError: Cannot read properties of null (reading 'testsuites')

Error while running tests:
TypeError: Cannot read properties of null (reading 'testsuites')

Finished running tests!

I suspect it's related to playwright, but that's just a guess.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.4
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Jedi"

linting
• flake8Path: "<placeholder>"
• mypyPath: "<placeholder>"
• pylintPath: "<placeholder>"

formatting
• autopep8Path: "<placeholder>"
• blackPath: "<placeholder>"

testing
• pytestEnabled: true

Extension version: 2023.14.0
VS Code version: Code 1.81.1 (Universal) (6c3e3dba23e8fadc360aed75ce363ba185c49794, 2023-08-09T22:20:33.924Z)
OS version: Darwin arm64 22.6.0
Modes:
Remote OS version: Linux arm64 5.15.49-linuxkit-pr

System Info

Item	Value
CPUs	Apple M1 Pro (10 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	4, 5, 5
Memory (System)	16.00GB (0.07GB free)
Process Argv	--crash-reporter-id 62586060-13a2-4d6f-9b3f-4b1de1d548e4
Screen Reader	no
VM	0%

Item	Value
Remote	Dev Container: azure_flask_postgres_flexible_appservice @ desktop-linux
OS	Linux arm64 5.15.49-linuxkit-pr
CPUs	unknown (5 x 0)
Memory (System)	8.74GB (4.89GB free)
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
vscccc:30803845
3biah626:30602489
f6dab269:30613381
a9j8j154:30646983
showlangstatbar:30737416
03d35959:30757346
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
copilotsettingt:30839829
synctok:30821570
dsvsc013:30795093
dsvsc014:30804076
diffeditorv1:30821571
dsvsc015:30829745

[eleanorjboyd]

Hello! Can you try this on the new rewrite testing experiment and see if that resolves your problem? You can do so by adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check for Experiment 'pythonTestAdapter' is active in your python logs.

Thanks!

[nsheaps]

Ignore this, I was running pytest with args, specifically pytest-xdists `-n 8` args, which breaks debugging. Pytest-xdist explicitly tells you that the `--pdb` option is disabled in the docs

Unfortunately having the same issue with the experiment enabled. Test runs fine when not debugging and I can debug a normal python script. In this case I'm running it directly from the test (right click > debug test)

Python 3.11.2
Extra details (including the requested opt-in experiment) below

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

experiments
• optInto: ["pythonTestAdapter"]

System Info

CPUs	Apple M2 Pro (12 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	4, 5, 5
Memory (System)	32.00GB (2.97GB free)
Process Argv	/Users/nathan.heaps/src/opensea-api --crash-reporter-id 954ffc0a-bf81-4dfe-beee-91e0cf8d0c29
Screen Reader	no
VM	0%

AB Test Info

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
89544117:30613380
a9j8j154:30646983
showlangstatbar:30737416
0bi6i642:30841073
a2ce3375:30757347
ecj1e332:30736112
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
copilotsettingt:30839829
asynctok:30821568
dsvsc013:30795093
dsvsc014:30804076
diffeditorv2:30821572
dsvsc015:30829745

[eleanorjboyd]

What is python.testing.pytestargs set to in your settings? Thinking it is related to: #21176

[nsheaps]

The value for me is:

  "python.testing.pytestArgs": [
    ".",
    // see: https://pytest-xdist.readthedocs.io/en/latest/distribution.html
    // NOTE: you cannot use multiple workers AND debug tests
    // see: https://pytest-xdist.readthedocs.io/en/latest/known-limitations.html#debugging
    "-n",
    "8",
    "--randomly-dont-reset-seed",
    "--disable-socket",
    "--allow-unix-socket",
    "--allow-hosts="
  ],

I gave it a shot with no args and it worked, sorry, didn't realize there were 2 threads for the same error. I'll add my details to the other thread. What's even better is I left that note for myself 2 weeks ago and forgot, sorry!

[pamelafox]

Hm, it says that setting isn't valid in a dev container:

I'll see if I can get the environment working locally.

[pamelafox]

Before I do that: I notice there's an error in the debug console when I try to debug a test:

Looks like you launched a headed browser without having a XServer running.                     Set either 'headless: true' or use 'xvfb-run <your-playwright-app>' before running Playwright. <3 Playwright Team                                                                                 

That's interesting because the default mode for playwright tests is headless, you have to specifically pass in the headed argument to run them headed. That makes me wonder if debug mode is somehow passing in an argument or setting an environment variable that makes them try to run in headed mode. Maybe the DISPLAY variable? I see a similar issue here:
microsoft/playwright-python#833

To clarify: It's already known that playwright tests can't run in headed mode inside a dev container, so the mystery is why debug mode prompts the headed mode.

[eleanorjboyd]

@pamelafox if you have the setting in your user settings it should copy over the remote settings- does that help? I will look at the playwright repo to see if I can repro this behavior. Thanks!

[eleanorjboyd]

Hello! Wanted to follow up, did this work if you moved the setting to your user settings?

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!