Indefinitely "Discovering Python Interpreters" due to faulty conda binary

[ablacklama]

Type: Bug

Behaviour

Expected vs. Actual

When i open a python project the language server never loads. It just stalls on "discovering python interpreters" forever.

In the output section i've put just one example of the errors ms-python is outputting. All the other errors are copies of the ones there but with different environment names and python versions.
It looks like it's searching for my old, deleted virtualenvs. I don't know where it's saving its information about those old envs or i'd delete that.

Steps to reproduce:

1. Open a python project with vscode-remote on my machine?

I'm sure it's something weird about my setup but i can't pin it down. I've reinstalled vscode, vscode-server, all extensions, ect.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.2
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): PipEnv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2023-09-21 14:16:59.993 [warning] Identifier for virt-virtualenv failed to identify /home/ablacklama/.virtualenvs/pyx_analysis-3.9/bin/python [Error: ENOENT: no such file or directory, scandir '/home/ablacklama/.virtualenvs/pyx_analysis-3.9/bin'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'scandir',
  path: '/home/ablacklama/.virtualenvs/pyx_analysis-3.9/bin'
}
2023-09-21 14:17:04.010 [error] Error: Command failed: /home/ablacklama/.virtualenvs/pyx_analysis-3.9/bin/python -I /home/ablacklama/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/get_output_via_markers.py /home/ablacklama/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/interpreterInfo.py
/bin/sh: 1: /home/ablacklama/.virtualenvs/pyx_analysis-3.9/bin/python: not found

    at ChildProcess.exithandler (node:child_process:420:12)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:489:12)
    at maybeClose (node:internal/child_process:1091:16)
    at Socket.<anonymous> (node:internal/child_process:449:11)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at Pipe.<anonymous> (node:net:322:12) {
  code: 127,
  killed: false,
  signal: null,
  cmd: '/home/ablacklama/.virtualenvs/pyx_analysis-3.9/bin/python -I /home/ablacklama/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/get_output_via_markers.py /home/ablacklama/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/interpreterInfo.py'
}

User Settings

languageServer: "Pylance"

formatting
• provider: "none"
• blackArgs: "<placeholder>"

testing
• promptToConfigure: false
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.16.0
VS Code version: Code 1.82.2 (abd2f3db4bdb28f9e95536dfa84d8479f1eb312d, 2023-09-14T05:55:25.390Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Remote OS version: Linux x64 5.15.0-83-generic

System Info

Item	Value
CPUs	AMD Ryzen 9 5900X 12-Core Processor (24 x 3700)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	31.94GB (12.57GB free)
Process Argv	--crash-reporter-id 0e2b899c-1a5d-43cb-be1d-1dada59d9714
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: cahserver
OS	Linux x64 5.15.0-83-generic
CPUs	AMD Ryzen Threadripper 3970X 32-Core Processor (64 x 3997)
Memory (System)	251.54GB (242.35GB free)
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
py29gd2263cf:30792227
vscaac:30438847
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
89544117:30613380
showlangstatbar:30737416
03d35959:30757346
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
copilotsettingc:30839828
synctok:30821570
dsvsc013:30795093
dsvsc014:30804076
diffeditorv2:30821572
dsvsc015cf:30829746
pythonfb280951cf:30830811

[karrtikr]

Try out the pre-release version of the extension:

You may need to clear cache using Python: Clear cache and reload command.

[ablacklama]

You may need to clear cache using Python: Clear cache and reload command.

@karrtikr
Switching to pre-release didn't do anything. Clearing cache got rid of all the environment failures. Now I'm left with this set of warnings/errors that were probably the true cause all along. It's progress, any other ideas?

2023-09-21 16:20:08.342 [warning] Retry failed, storage update failed for key PYTHON_GLOBAL_STORAGE_KEYS
2023-09-21 16:20:08.342 [warning] Retry failed, storage update failed for key PYTHON_GLOBAL_STORAGE_KEYS
2023-09-21 16:20:08.342 [warning] Retry failed, storage update failed for key PYTHON_GLOBAL_STORAGE_KEYS
2023-09-21 16:20:08.342 [warning] Retry failed, storage update failed for key PYTHON_GLOBAL_STORAGE_KEYS
2023-09-21 16:20:08.343 [warning] Retry failed, storage update failed for key PYTHON_WORKSPACE_STORAGE_KEYS
2023-09-21 16:20:08.343 [warning] Retry failed, storage update failed for key PYTHON_WORKSPACE_STORAGE_KEYS
2023-09-21 16:20:08.343 [warning] Retry failed, storage update failed for key PYTHON_WORKSPACE_STORAGE_KEYS
2023-09-21 16:20:54.231 [error] Autoselection timeout out, this is likely a issue with how consumer called execution factory API. Using default python to execute.
2023-09-21 16:20:54.274 [info] > python -c "import sys;print(sys.executable)"
2023-09-21 16:20:54.359 [info] Python interpreter path: ~/.pyenv/versions/3.10.5/bin/python
2023-09-21 16:20:56.423 [error] Autoselection timeout out, this is likely a issue with how consumer called execution factory API. Using default python to execute.
2023-09-21 16:20:56.430 [error] Autoselection timeout out, this is likely a issue with how consumer called execution factory API. Using default python to execute.
2023-09-21 16:20:56.873 [error] Autoselection timeout out, this is likely a issue with how consumer called execution factory API. Using default python to execute.

[karrtikr]

When i open a python project the language server never loads

The intention of pre-release was to help with this. We just made a fix to ensure Interpreter discovery to not block LS: #22041, so even if Discovery does not finish, LS should ideally start, as long as you see an active interpreter on the bottom right:

Can you confirm whether you see it? Your log seems to indicate: Python interpreter path: ~/.pyenv/versions/3.10.5/bin/python, which suggests you do.

[ablacklama]

it waits for about a full minute (where i don't see an active interpreter) then selects an interpreter.

However, code completion is broken even after an active interpreter is selected.

[ablacklama]

Oh and I made a mistake when i said clearing the cache fixed the environment errors. They just took longer to show up after clearing the cache. It's still throwing errors while looking for the same deleted environments.

[ablacklama]

I'll also remind you that i'm using vscode remote ssh'd into an ubuntu box. Since that has often been a factor in previous issues i've had.

[ablacklama]

Oh i seem to have fixed code completion for now. I opened vscode without being ssh'd into the linux box and cleared the cache. That seems to have worked. I almost never open vscode without using vscode-remote so i hadn't tried that.

Now it's just stuck on "Refreshing Python Interpreters" but at least I can ignore that and still work effectively.

[karrtikr]

Great, now we can work on making that go away. For that I would need the complete debug logs:

• Run Developer: Set log level command
• Select Python -> Select Trace
• Run Developer: Reload Window command
• Provide the output:

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

[karrtikr]

It's still throwing errors while looking for the same deleted environments.

It's likely that other extensions like Jupyter are still looking for those deleted environments, in which you can try disabling the extension. The logs should contain this info as well.

[ablacklama]

Update: It keeps breaking in the same way and i keep having to use a non vscode-remote ssh'd window to clear the python cache in order to make code completion work for a few hours.

[karrtikr]

I noticed from the log that we run the

~/miniconda3/bin/conda info --json

command which never returns back. Can you look into it, and if that file event exists?

[karrtikr]

While I know it's not ideal, if you install conda on your Remote Linux machine, it can make the issue go away:

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

[ablacklama]

Oh that looks like it worked! I just deleted the miniconda3 folder entirely since i wasn't using it and it's no longer stuck on discovering interpreters. Thanks a lot!

As a side note it still looks for old environments that don't exist, but at least that's not causing any issues on my side. Just some errors in the logs.

[karrtikr]

Awesome. Happy to help unblock you.

In the logs you sent me I don't see any, but let us know if that causes any trouble in the future.

[karrtikr]

By any chance do you know if ~/miniconda3/bin/conda actually existed in your system prior to the delete?

[ablacklama]

Yes ~/miniconda3/bin/conda did exist. Out of curiosity I tried calling ~/miniconda3/bin/conda info --json it manually and it just hung and never returned.

[peterwaller-arm]

It appears the issue I described above is fixed on the pre-release pylance. Sorry for the noise.

[karrtikr]

No worries, and thanks for all the help 🙂 I've created a fix to bypass such problematic conda binaries on the system, ensuring discovery can complete even when they're present: #22072.