[BUG] multiple lsp_server.py processes are spawned for each folder in a multi-root workspace

[tboddyspargo]

I've been trying to troubleshoot some slowness from VSCode and have followed clues that have led me to some observations about the pylint extension. I'm hoping for some clarity on the behavior or perhaps some reassurance that there is a bug.

When I go to Help > Open Process Explorer I see two pylint subprocess families, each of which contains sub-nodes (sub-processes?) for every folder in my multi-root workspace.

I only expected one process to be running per folder in my workspace, so this felt like a likely source of duplicative work.

NOTE: My .code-workspace file contains one folder for the root ("path": ".") and then folders for every package ("path": "pkg-a") - of which there are quite a few.

redacted `.code-workspace` file

{
  "folders": [
    {
      "path": "."
    },
    {
      "path": "pkg-a"
    },
  ],
  "settings": {
    "python.languageServer": "Pylance",
    // "python.defaultInterpreterPath": ".venv/bin/python" is defined per-component in [dir]/.vscode/settings.json
    "pylint.args": [
      "--rcfile=${workspaceFolder:datachat}/.pylintrc", // shared pylint config for all packages.
    ],
}

redacted root `.vscode/settings.json` file

{
  "python.defaultInterpreterPath": "pyenv exec python",
  "python.analysis.include": [
    "fake" // disable analysis at the root folder, since it will be handled for all relevant sub-folders.
  ],
  "python.testing.pytestEnabled": false,
  // NOTE: I'd like to do something like `"pylint.enable": false` here to disable the extension in specific sub-folders, but don't see a way to do that...
}

redacted `pkg-a/.vscode/settings.json` file

{
  "python.testing.pytestEnabled": true,
  "python.defaultInterpreterPath": ".venv/bin/python",
}

The logs seem to support the fact that two identical pylint lsp_server.py processes are running. Everything is logged twice in rapid succession (full logs below):

2023-12-13 09:47:09.023 [info] Server run command: /Users/tyler/.pyenv/versions/3.11.6/bin/python /Users/tyler/.vscode/extensions/ms-python.pylint-2023.10.1/bundled/tool/lsp_server.py
2023-12-13 09:47:09.029 [info] Server: Start requested.
2023-12-13 09:47:09.029 [info] Server run command: /Users/tyler/.pyenv/versions/3.11.6/bin/python /Users/tyler/.vscode/extensions/ms-python.pylint-2023.10.1/bundled/tool/lsp_server.py
2023-12-13 09:47:09.029 [info] Server: Start requested.
2023-12-13 09:47:10.212 [info] CWD Server: /Users/tyler/dev/workspace
2023-12-13 09:47:10.212 [info] CWD Server: /Users/tyler/dev/workspace

Full pylint extension startup log

Questions

• Why does the pylint extension have two process families? Are they redundant? If so, is the current behavior a "bug"?
• Is there a way (or can it be added) to disable the pylint extension for specific folders in a multi-root workspace?
  • I'm happy to add a separate issue for this, if it would amount to a feature request.
• Is there a work-around that would allow me to achieve a similar result as "pylint.enable": false?
  • e.g. pylint will actually run for the folder, but it would exit almost immediately without doing any work (or something to that effect).
• If my multi-root workspace has both the root "." and a subdirectoy "./pkg-a" and I save changes to ./pkg-a/main.py, will two separate pylint processes perform the linting, or will the extension avoid that redundancy?
  • What's the best way to avoid this redundancy (if any)?

[tboddyspargo]

• Why does the pylint extension have two process families? Are they redundant? If so, is the current behavior a "bug"?

FWIW - I tried the pre-release version and I now only see a single process family. Maybe this part has already been addressed... confirmation that that was an identified issue and has been intentionally addressed would be great.

UPDATE:: Nevermind, the issue is still present in the pre-release version. In fact, certain actions seem to trigger spawning ADDITIONAL pylint process families, multiplying the problem.

I'd still love some input on my other questions, but I can open a separate issue if that would help.

Thanks in advance!

[tboddyspargo]

UPDATE:: Nevermind, the issue is still present in the pre-release version. In fact, certain actions seem to trigger spawning ADDITIONAL pylint process families, multiplying the problem.

Steps I took to provoke this (not sure which might be relevant or superfluous):

1. Use VSCode VCS branch pane to switch to a different branch
2. Remove "pylint.args" setting from .code-workspace and move it to a specific folder's .vscode/settings.json file (and save those changes)
3. Open a __init__.py file from a package directoy and save it.

There now seem to be anywhere from 4-6 pylint processes for every folder:

2023-12-13 11:33:09.238 [info] /opt/homebrew/bin/python3 -m pylint --reports=n --output-format=json --version --clear-cache-post-run=y --from-stdin /Users/tyler/dev/workspace/pkg-a/web-server/migrations/__init__.py
2023-12-13 11:33:09.238 [info] /opt/homebrew/bin/python3 -m pylint --reports=n --output-format=json --version --clear-cache-post-run=y --from-stdin /Users/tyler/dev/workspace/pkg-a/web-server/migrations/__init__.py
2023-12-13 11:33:09.238 [info] /opt/homebrew/bin/python3 -m pylint --reports=n --output-format=json --version --clear-cache-post-run=y --from-stdin /Users/tyler/dev/workspace/pkg-a/web-server/migrations/__init__.py
2023-12-13 11:33:09.238 [info] /opt/homebrew/bin/python3 -m pylint --reports=n --output-format=json --version --clear-cache-post-run=y --from-stdin /Users/tyler/dev/workspace/pkg-a/web-server/migrations/__init__.py
2023-12-13 11:33:09.239 [info] CWD Linter: /Users/tyler/dev/workspace/pkg-a
2023-12-13 11:33:09.239 [info] CWD Linter: /Users/tyler/dev/workspace/pkg-a
2023-12-13 11:33:09.239 [info] CWD Linter: /Users/tyler/dev/workspace/pkg-a
2023-12-13 11:33:09.239 [info] CWD Linter: /Users/tyler/dev/workspace/pkg-a

I'm guessing that whatever is causing this, could be a contributor to #454, also. Just a hunch.

[tboddyspargo]

Is there a work-around that would allow me to achieve a similar result as "pylint.enable": false?

• e.g. pylint will actually run for the folder, but it would exit almost immediately without doing any work (or something to that effect).

FWIW, I tried --disable=all, but then pylint emits non-json output (No files to lint: exiting.) and causes a JSON parsing error from the extension. The extension should probably gracefully handle this case (probably by checking the exit code).

[tboddyspargo]

1. Use VSCode VCS branch pane to switch to a different branch

Switching branches seems to be a somewhat reliable reproduction for me it ends up spawning redundant pylint process families for the extension.

• Pytlint: Restart Server doesn't eliminate the redundant process families.
• My only current work-around is to close and re-open VSCode.

[karthiknadig]

@tboddyspargo Can you elaborate on the actions that seem to trigger this?

Why does the pylint extension have two process families? Are they redundant? If so, is the current behavior a "bug"?

There should be 1 process family per VS Code instance. lsp_server is the main coordinator, and it will spawn processes under it depending on the python selected for the various folders in the multi-root case. The sub processes will be using lsp_runner. Basically if there is a settings change, then it triggers launching of these processes. But, the life-time is controlled by VS Code.

I think in your case moving between branches, is changing some setting that is triggering this.

Is there a way (or can it be added) to disable the pylint extension for specific folders in a multi-root workspace? I'm happy to add a separate issue for this, if it would amount to a feature request.

I just merged this feature. It will be in pre-release tomorrow. Basically pylint.enabled: false would turn it off for that workspace folder.

Is there a work-around that would allow me to achieve a similar result as "pylint.enable": false? e.g. pylint will actually run for the folder, but it would exit almost immediately without doing any work (or something to that effect).

See above answer. PR here #475

If my multi-root workspace has both the root "." and a subdirectoy "./pkg-a" and I save changes to ./pkg-a/main.py, will two separate pylint processes perform the lining, or will the extension avoid that redundancy? What's the best way to avoid this redundancy (if any)?

It will (should) only run under ./pkg-a. We try to go from the file to the nearest parent that is a workspace folder.

[karthiknadig]

The issue with #454 is different. ruff and pylint use the same server base. ruff is able to manage with lint on change, but pylint is not, and the blocker here is due to time it takes to process all the queued requests. ruff run fast enough that the queue gets cleared immediately.

The solution there is to switch to a mode where the IDE requests for the problems rather than the LSP server reporting problems on open, save, or close. That way, IDE has more control over when such request needs to be raised, and it could control cancellation of those requests.

[tboddyspargo]

Thanks for your responses, @karthiknadig! I'm excited to hear that enabled configuration option will be arriving soon.

@tboddyspargo Can you elaborate on the actions that seem to trigger this?

Why does the pylint extension have two process families? Are they redundant? If so, is the current behavior a "bug"?

... Basically if there is a settings change, then it triggers launching of these processes. But, the life-time is controlled by VS Code.

I think in your case moving between branches, is changing some setting that is triggering this.

As I think about which settings changes might be coming along as part of the branch change, I examined a diff of the recent branch change I did (corresponding to my screenshot above). The only changes I see are:

• Added a new folder to .code-workspace file
• Added a corresponding .vscode/settings.json file in that folder (matching the pkg-a/.vscode/settings.json example from above).

Could changes to the members of the folders key in a .code-workspace file trigger this kind of process duplication? In my examples, the number of additional processes doesn't really seem to correspond to the number of added folders, but there could be other contributing factors.

FWIW, when I just make manual modifications to the .code-workspace file or any settings.json file in my multi-root workspace (no branch changes), I do see the single pylint extension parent process restarting all of its folder-specific subprocesses without any extraneous process families being spawned.

[tboddyspargo]

As I think about which settings changes might be coming along as part of the branch change, I examined a diff of the recent branch change I did (corresponding to my screenshot above). The only changes I see are:

• Added a new folder to .code-workspace file
• Added a corresponding .vscode/settings.json file in that folder (matching the pkg-a/.vscode/settings.json example from above).

I just restarted VSCode and then performed another branch switch in which non-pylint config changes were made to .code-workspace and the root .vscode/settings.json file. Two new pylint process families were spawned (bringing total up to 3, each with subprocesses for every workspace folder).

Changes in the files included:

• changing to "editor.defaultFormatter": "ms-python.black-formatter" and "editor.formatOnSaveMode": "file"
• Adding "ruff.lint.args"
• Removing some "python.analysis" options
• Adding "python.testing.pytestEnabled": false, to the root settings.json file

My folder theory doesn't seem to hold water.

Next, and without restarting VSCode, I switched to another branch that contained no .code-workspace or settings.json differences with the previous branch. An alarming number of process families were spawned and VSCode became quite sluggish: