Highlighting fails after switching between monorepo projects (when switching python interpreters)

[alita-moore]

Applies To

• Notebooks (.ipynb files)
• Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?

Basically, I expect the highlighting on the Jupyter notebook to update when I switch from one project to another. However, when I switch from one project to another it fails to update the highlighting until I either reload the window or I restart the extension host. Note that normal python files highlight just fine.

To reproduce this see the following codesandbox

After you've loaded into the codesandbox, open it in vscode. Make sure that you have the following extensions:

1. pylance
2. jupyter
3. python envy (teticio.python-envy)

open "third_project/another_one.ipynb" and select the appropriate .venv which is located at "third_project/.venv". Notice that things are highlighting.

open other_project/init.py

You can also get here by navigating to the definition of the "other_project" module. Notice that python envy changes the interpreter.

Now navigate back to the "third_project/another_one.ipynb". If you're "lucky" you'll see that when you scroll down and then back up (to force re-draw the highlihgts) that sometimes it freezes. When it freezes like this it's frozen indefinitely.

NOTE:

1. It may be somewhat difficult to reproduce the results, and you may have to go back and forth between notebooks / files quickly in order to overload the system and then force the behavior. However, note that in larger codebases it happens nearly every time you switch between notebooks / project interpreters.
2. I am using poetry to import from other modules in the monorepo using their relative path importing system

VS Code Version

1.90.0

Jupyter Extension Version

v2024.5.0

Jupyter logs

Visual Studio Code (1.90.0, ssh-remote, desktop)
Jupyter Extension Version: 2024.5.0.
Python Extension Version: 2024.8.0.
Pylance Extension Version: 2024.6.1.
Platform: linux (x64).
Workspace folder /project/workspace, Home = /root
14:55:37.492 [info] Starting Kernel (Python Path: /project/workspace/third_project/.venv/bin/python, Poetry, 3.8.19) for '/project/workspace/third_project/another_one.ipynb' (disableUI=true)
14:55:38.300 [info] Process Execution: /project/workspace/third_project/.venv/bin/python -m pip list
14:55:38.367 [info] Process Execution: /project/workspace/third_project/.venv/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
14:55:38.382 [info] Process Execution: /project/workspace/third_project/.venv/bin/python -m ipykernel_launcher --f=/~/.local/share/jupyter/runtime/kernel-v2-18791fQR1MB8Io0Zd.json
    > cwd: //project/workspace/third_project
14:55:40.250 [info] Kernel successfully started
14:55:40.270 [info] Process Execution: /project/workspace/third_project/.venv/bin/python /~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/printJupyterDataDir.py
14:58:47.202 [info] Starting Kernel (Python Path: /project/workspace/.venv/bin/python, Poetry, 3.8.19) for '/project/workspace/notebook.ipynb' (disableUI=true)
14:58:48.116 [info] Process Execution: /project/workspace/.venv/bin/python -m pip list
14:58:48.189 [info] Process Execution: /project/workspace/.venv/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
14:58:48.206 [info] Process Execution: /project/workspace/.venv/bin/python -m ipykernel_launcher --f=/~/.local/share/jupyter/runtime/kernel-v2-18791DaYZEb5bjEdQ.json
    > cwd: //project/workspace
14:58:50.302 [info] Kernel successfully started
14:58:50.317 [info] Process Execution: /project/workspace/.venv/bin/python /~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/printJupyterDataDir.py
14:58:57.241 [info] Starting Kernel (Python Path: /project/workspace/other_project/.venv/bin/python, Poetry, 3.8.19) for '/project/workspace/other_project/other.ipynb' (disableUI=true)
14:58:57.949 [info] Process Execution: /project/workspace/other_project/.venv/bin/python -m pip list
14:58:58.162 [info] Process Execution: /project/workspace/other_project/.venv/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
14:58:58.192 [info] Process Execution: /project/workspace/other_project/.venv/bin/python -m ipykernel_launcher --f=/~/.local/share/jupyter/runtime/kernel-v2-18791UmM5kZOsfZ8o.json
    > cwd: //project/workspace/other_project
14:59:00.672 [info] Kernel successfully started
14:59:00.691 [info] Process Execution: /project/workspace/other_project/.venv/bin/python /~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/printJupyterDataDir.py

Coding Language and Runtime Version

python v3.8

Language Extension Version (if applicable)

Pylance v2024.6.1

Anaconda Version (if applicable)

No response

Running Jupyter locally or remotely?

Remote

[alita-moore]

I think it has something to do with pylance not finishing loading with one module before switching to another. I found that if I switch back to the previous module and then wait for the highlights to paint that it sometimes unsticks the jupyter notebook.

i.e. starting at notebook a in module A going to file b in module B, if you go back to a too quickly then it seems to get stuck. But if you navigate back to b and wait for a little bit, it sometimes allows a to paint as well.

NOTE: this only occurs when you remain on b and have a open in a separate tab / split screen. However, if you navigate back to a it remains stuck.

[alita-moore]

Also, I've noticed that my notebook stops saving properly when this happens as well. It claims that it's failed to save and no matter how many times I retry it fails to save. This is very frustrating because it causes me to lose progress. I'm not sure if that's a different problem associated with restarting the extension host, though.

The only way to save my notebook is to save it to a different filename and then manually overwrite the previous ipynb :(

The only way to stop this behavior is to rename the ipynb

[alita-moore]

Another behavior I noticed was that if the highlighting is frozen and you change the project interpreter manually (i.e. cmd + shift + p -> select interpreter) it unfreezes the highlights. But then if you go back to the interpreter that's the same as one being used for the jupyter kernel it will then freeze again.

[alita-moore]

Here's a video of me reproducing the problem, it took me a few tries but you'll see near the end that it freezes as described:
https://www.loom.com/share/d7cb8b8ec70942998904e5d6f4e9c6f6?sid=4ea8ca7d-972f-4274-97e3-cb1fecdb176c

[debonte]

Since Pylance owns semantic highlighting for Python, I've brought the main issue over to our repo and will resolve #5986 as a dupe.

[debonte]

Here's the link to your log file which was the only piece of info that I think was specific to the other issue:

Here are the logs of a time when I think I reproduced it.
logs.txt

I'd be happy to collect more if you want.

[alita-moore]

I've noticed that this seems to happen more often when the setting python.analysis.nodeExecutable is set to \usr\bin\node (or something other than empty). I started noticing this after I changed this setting due to memory overload warnings. note that it still happens, though.

[alita-moore]

One work-around I have found is to disable then enable the jupyter extension without reloading vscode. In other words, disable without committing that change. I think this problem is related to this post as well: https://stackoverflow.com/questions/78643088/pylance-in-vscode-jupiter-notebooks-cannot-resolve-modules

[alita-moore]

force-stopping the pylance processes also causes it to get unstuck. Is there a way to restart pylance every time the interpreter changes? And or can you add a command to restart all processes? It would make my life a lot easier :)

[debonte]

Is there a way to restart pylance every time the interpreter changes?

No. It shouldn't be necessary to restart the process. Pylance will be notified of the config change.

And or can you add a command to restart all processes?

That exists already actually -- "Python: Restart Language Server"

[debonte]

I haven't been able to repro this. Every time I think it has repro'd, I wait a little bit and it recovers. I'm curious if the same would happen for you if you waited longer. I'm not saying that that's a good experience -- but if it does recover eventually that would be good to know. It would also be interesting to see what's happening in the "Python Language Server" log (in the Output pane) when you believe that this has repro'd. Is it actively doing anything? Is it still catching up on recent interpreter changes triggered by Python Envy?

Changing the interepreter "every" time you change tabs can be expensive as it causes us to clear our type cache (we need to start from scratch on analysis), reindex your files, etc. So I'm not surprised that you see more issues in a real monorepo with more files.

Have you tried opening your repo as a multiroot workspace rather than simply opening the root folder? You'd use "Add Folder to Workspace" to add each folder that has its own venv. That's a one time thing. You'd then save the layout as a .code-workspace file that you can open (instead of the root folder) in the future. At that point you should disable Python Envy as Pylance wouldn't need its help to identify each file's corresponding venv. And switching between tabs in different projects wouldn't require an expensive configuration change.

Of course, that's assuming that you're not relying on Python Envy for other things.

@heejaechang might have other ideas.

[alita-moore]

I'll look into the .code-workspace I haven't heard of that before but am definitely open to a smoother experience! The language server output does spend some time catching up but eventually it stops outputting and it never recovers. I think it's possible that if given enough time it could recover, but I also find that unlikely because I have gone hours without it recovering. I have also noticed the behavior you mentioned, though. Especially in the sample monorepo I provided, it's possible that's not big enough to trigger the actual behavior. In that case it likely makes the most sense for me to just share logs from my project with you (not publicly).

By the way, I've noticed that when it freezes some pylance errors stop functioning correctly. Like it won't error on undefined variables until the processes are manually reset even in python files in different workspace roots and when highlighting is working for those files.

That exists already actually -- "Python: Restart Language Server"

I don't see this command, but will try to find it as that would be helpful for me to quickly refresh / get highlighting to work.

What other information / logs can I collect to you and where I can securely share it?

[alita-moore]

I've looked into the workspaces that you mentioned. I'll keep messing around with it to try and make it work for my use-case, but I'll point out that python-envy is more intuitive and requires less setup.

update: played around with it more, seems interesting, will let you know if I run into issues

[debonte]

That exists already actually -- "Python: Restart Language Server"

I don't see this command, but will try to find it as that would be helpful for me to quickly refresh / get highlighting to work.

That's strange. You're looking in the Command Palette? Ctrl+Shift+P?

I'll look into the .code-workspace I haven't heard of that before

Here is the docs page on that: https://code.visualstudio.com/docs/editor/multi-root-workspaces

it likely makes the most sense for me to just share logs from my project with you (not publicly).

You can email me at erikd at microsoft dot com. You could send me a zip file, or a link to a download location of your choosing. Please set "python.trace.server": "verbose" which will cause the LSP traffic between VS Code and Pylance to be logged. I'm curious to see the last semanticTokens request and response.

I'll point out that python-envy is more intuitive and requires less setup.

That's fair. We're aware that our monorepo support is a bit rough at the moment.

[alita-moore]

Ah, one problem I'm seeing with the workspaces solution is that you can't refactor across project roots. When using python envy refactors propogate

[debonte]

Ah, one problem I'm seeing with the workspaces solution is that you can't refactor across project roots. When using python envy refactors propogate

Yeah, it would have that limitation.

So I guess that means that you have cross-root/cross-project imports? Out of curiosity, how do you currently handle those imports -- are the paths absolute or relative? Are you happy with whatever approach you're using at the moment?

@erictraut, do you have any suggestions on how to handle a monorepo where the projects each have their own venv? Do we have a good way to handle that while still allowing refactoring (ex. rename) across projects? Would execution environments help here?

[alita-moore]

I sent the logs @debonte I didn't include much of the logs for security reasons, but I captured the final parts where it eventually stopped outputting after claiming that SemanticTokens full for the notebooks in question.

Yep, I have cross-root/cross-project imports using poetry development paths. This affords me a similar workflow to typescript monorepos which is very nice. Basically, it lets me check the usage of some variable / function whatever across the entire monorepo and refactor it as well as if it were a single project. Currently I setup my project in the same way as the original codesandbox that I linked, I setup each project in the same structure as a typescript monorepo (with pnpm as well for added control) and make sure that poetry saves the dependencies in the local .venv. Then when I navigate between projects python envy changes the interpreter which allows me to navigate naturally through the project.

I'm very happy with this workflow, the biggest issue being pylance being a bit delayed when switching and it using up a lot of CPU. But to me it's worth it because of the ease of refactoring and seeing where things are being used acros projects.

p.s. for reference here are the logs after the problem occurred and I was navigating through the frozen notebook:

2024-06-21 05:48:15.195 [info] (59718) [BG(1)] SemanticTokens full at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W1sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:15.371 [info] (59718) [BG(1)] SemanticTokens range 0:0 - 3:65 at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W1sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:25.582 [info] (59718) [BG(1)] SemanticTokens full at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W4sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:25.663 [info] (59718) [BG(1)] SemanticTokens full at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W5sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:26.062 [info] (59718) [BG(1)] SemanticTokens range 0:0 - 0:79 at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W4sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:26.128 [info] (59718) [BG(1)] SemanticTokens range 0:0 - 31:28 at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W5sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:27.048 [info] (59718) [BG(1)] SemanticTokens full at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W6sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:27.488 [info] (59718) [BG(1)] SemanticTokens range 0:0 - 11:39 at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#W6sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:28.003 [info] (59718) [BG(1)] SemanticTokens full at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#X10sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)
2024-06-21 05:48:28.412 [info] (59718) [BG(1)] SemanticTokens range 0:0 - 102:0 at file:///workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb.py#X10sdnNjb2RlLXJlbW90ZQ%3D%3D (0ms)

with python trace on:

2024-06-21 05:55:32.847 [info] [Trace - 5:55:32 AM] Sending request 'textDocument/codeAction - (598)'.
2024-06-21 05:55:32.847 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 0,
            "character": 0
        }
    },
    "context": {
        "diagnostics": [],
        "triggerKind": 2
    }
}


2024-06-21 05:55:32.847 [info] [Trace - 5:55:32 AM] Sending request 'textDocument/codeAction - (599)'.
2024-06-21 05:55:32.847 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X11sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 0,
            "character": 0
        }
    },
    "context": {
        "diagnostics": [],
        "triggerKind": 2
    }
}


2024-06-21 05:55:32.849 [info] [Trace - 5:55:32 AM] Received response 'textDocument/codeAction - (597)' in 2ms.
2024-06-21 05:55:32.849 [info] Result: []


2024-06-21 05:55:32.849 [info] [Trace - 5:55:32 AM] Received response 'textDocument/codeAction - (598)' in 2ms.
2024-06-21 05:55:32.849 [info] Result: []


2024-06-21 05:55:32.849 [info] [Trace - 5:55:32 AM] Received response 'textDocument/codeAction - (599)' in 2ms.
2024-06-21 05:55:32.849 [info] Result: []


2024-06-21 05:55:33.233 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/semanticTokens/full - (600)'.
2024-06-21 05:55:33.233 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X13sdnNjb2RlLXJlbW90ZQ%3D%3D"
    }
}


2024-06-21 05:55:33.233 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/semanticTokens/full - (601)'.
2024-06-21 05:55:33.233 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D"
    }
}


2024-06-21 05:55:33.233 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/semanticTokens/full - (602)'.
2024-06-21 05:55:33.233 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X11sdnNjb2RlLXJlbW90ZQ%3D%3D"
    }
}


2024-06-21 05:55:33.233 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/inlayHint - (603)'.
2024-06-21 05:55:33.233 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X13sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 11,
            "character": 14
        }
    }
}


2024-06-21 05:55:33.233 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/inlayHint - (604)'.
2024-06-21 05:55:33.233 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 20,
            "character": 57
        }
    }
}


2024-06-21 05:55:33.233 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/inlayHint - (605)'.
2024-06-21 05:55:33.233 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X11sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 9,
            "character": 74
        }
    }
}


2024-06-21 05:55:33.235 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/full - (600)' in 2ms.
2024-06-21 05:55:33.235 [info] Result: {
    "data": []
}


2024-06-21 05:55:33.235 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/full - (601)' in 2ms.
2024-06-21 05:55:33.235 [info] Result: {
    "data": []
}


2024-06-21 05:55:33.236 [info] [Trace - 5:55:33 AM] Received response 'textDocument/inlayHint - (603)' in 2ms.
2024-06-21 05:55:33.236 [info] No result returned.


2024-06-21 05:55:33.236 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/full - (602)' in 3ms.
2024-06-21 05:55:33.236 [info] Result: {
    "data": []
}


2024-06-21 05:55:33.236 [info] [Trace - 5:55:33 AM] Received response 'textDocument/inlayHint - (604)' in 3ms.
2024-06-21 05:55:33.236 [info] No result returned.


2024-06-21 05:55:33.236 [info] [Trace - 5:55:33 AM] Received response 'textDocument/inlayHint - (605)' in 3ms.
2024-06-21 05:55:33.236 [info] No result returned.


2024-06-21 05:55:33.319 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/semanticTokens/range - (606)'.
2024-06-21 05:55:33.319 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X13sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 11,
            "character": 14
        }
    }
}


2024-06-21 05:55:33.320 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/semanticTokens/range - (607)'.
2024-06-21 05:55:33.320 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 20,
            "character": 57
        }
    }
}


2024-06-21 05:55:33.320 [info] [Trace - 5:55:33 AM] Sending request 'textDocument/semanticTokens/range - (608)'.
2024-06-21 05:55:33.320 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X11sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 0,
            "character": 0
        },
        "end": {
            "line": 9,
            "character": 74
        }
    }
}


2024-06-21 05:55:33.321 [info] [Trace - 5:55:33 AM] Received notification 'telemetry/event'.
2024-06-21 05:55:33.321 [info] Params: {
    "Properties": {
        "lsVersion": "2024.6.101",
        "method": "textDocument.semanticTokens.range"
    },
    "Measurements": {
        "duration": 1
    },
    "EventName": "language_server/server_side_request"
}


2024-06-21 05:55:33.322 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/range - (606)' in 3ms.
2024-06-21 05:55:33.322 [info] Result: {
    "data": []
}


2024-06-21 05:55:33.322 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/range - (607)' in 2ms.
2024-06-21 05:55:33.322 [info] Result: {
    "data": []
}


2024-06-21 05:55:33.322 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/range - (608)' in 2ms.
2024-06-21 05:55:33.322 [info] Result: {
    "data": []
}


2024-06-21 05:55:34.041 [info] [Trace - 5:55:34 AM] Sending request 'textDocument/codeAction - (609)'.
2024-06-21 05:55:34.041 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D"
    },
    "range": {
        "start": {
            "line": 16,
            "character": 32
        },
        "end": {
            "line": 16,
            "character": 32
        }
    },
    "context": {
        "diagnostics": [],
        "triggerKind": 2
    }
}


2024-06-21 05:55:34.042 [info] [Trace - 5:55:34 AM] Received response 'textDocument/codeAction - (609)' in 1ms.
2024-06-21 05:55:34.042 [info] Result: []


2024-06-21 05:55:49.773 [info] [Trace - 5:55:49 AM] Sending request 'textDocument/foldingRange - (610)'.
2024-06-21 05:55:49.773 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X13sdnNjb2RlLXJlbW90ZQ%3D%3D"
    }
}


2024-06-21 05:55:49.773 [info] [Trace - 5:55:49 AM] Sending request 'textDocument/foldingRange - (611)'.
2024-06-21 05:55:49.773 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D"
    }
}


2024-06-21 05:55:49.774 [info] [Trace - 5:55:49 AM] Sending request 'textDocument/foldingRange - (612)'.
2024-06-21 05:55:49.774 [info] Params: {
    "textDocument": {
        "uri": "vscode-notebook-cell://codespaces%2Bominous-adventure-qv66jp65j7wfqp7/workspaces/data-engine/notebooks/main_pipeline%20copy.ipynb#X11sdnNjb2RlLXJlbW90ZQ%3D%3D"
    }
}


2024-06-21 05:55:49.774 [info] [Trace - 5:55:49 AM] Received response 'textDocument/foldingRange - (610)' in 1ms.
2024-06-21 05:55:49.774 [info] Result: []


2024-06-21 05:55:49.774 [info] [Trace - 5:55:49 AM] Received response 'textDocument/foldingRange - (611)' in 1ms.
2024-06-21 05:55:49.774 [info] Result: []


2024-06-21 05:55:49.775 [info] [Trace - 5:55:49 AM] Received response 'textDocument/foldingRange - (612)' in 1ms.
2024-06-21 05:55:49.775 [info] Result: []

[alita-moore]

To be clear btw poetry dev paths treats the other projects as packages. I'm the same way that you can import a pnpm package in Typescript

[erictraut]

do you have any suggestions on how to handle a monorepo where the projects each have their own venv

These are the options I can think of:

1. Build a single venv that is the superset of all of the venvs used across the monorepo, and use that for editing, refactoring, and type checking. This is the technique I've used previously for a monorepo. It requires that the libraries installed in the various venvs are not too divergent (e.g. conflicting versions of the same library).
2. Use workspaces. This gives you per-venv isolation and analysis, but it also means that refactoring doesn't work across workspaces.

[debonte]

2024-06-21 05:55:33.235 [info] [Trace - 5:55:33 AM] Received response 'textDocument/semanticTokens/full - (600)' in 2ms.
2024-06-21 05:55:33.235 [info] Result: {
"data": []
}

Given the empty semantic tokens response, my best guess is that we're somehow no longer aware that the cells in that notebook exist. How we got into that state, I'm not sure. I'll try to repro this some more.

[alita-moore]

@erictraut that's not an ideal solution because it causes conflicts when there are some projects that require heavy dependencies or have conflicts with others (torch being a good example) . Have you found this to be an issue as well? Using python envy + poetry dev paths allows me to have project configs for each project that are separate from each other in the same way you'd have a monorepo in typescript. Have you given that a try?

[erictraut]

Have you found this to be an issue as well?

We tried to keep versions of third-party libraries roughly in sync across the monorepo, so this wasn't an issue. In cases where the library version wasn't exactly in sync, we were typically able to use the later version and assume that the earlier version was backward compatible. In a few rare cases, we needed to create custom stubs.

that's not an ideal solution

I don't think an ideal solution is possible in this case. You're going to need to make some compromises. If your monorepo is using different venvs, then any modules that are used in two or more venvs are potentially going to be interpreted differently because of symbols they import from the different venvs. Even if pylance were to understand different venvs without using workspaces (something that's very unlikely because it would require a substantial redesign and rewrite of pylance and pyright), it would still be dangerous to perform refactoring actions across venv boundaries. Refactoring is based on static type information, and the venv affects the static analysis. That means a refactor action that is correct for one venv may not be for another. Most of the time this probably wouldn't matter, and the refactor would be safe, but the edge cases would be ugly.

[alita-moore]

@erictraut just out of curiosity, how do you deal with submodules and dependencies that have dependency conflicts that cross major version updates?

I see, would it also require a lot of work to make switching between the venv more efficient / fast?

[erictraut]

would it also require a lot of work to make switching between the venv more efficient / fast?

Are you finding this to be slow currently? Pyright alone (without the additional functionality in pylance) should enable very fast switching between venvs. Pylance may add overhead if it needs to reindex based on the new venv, but indexing is done in the background.

[debonte]

Are you finding this to be slow currently?

Semantic highlighting is slow, and that's also on the background thread. But the real issue is that sometimes, after a venv change, semantic highlighting stops working entirely. I suspect that that's happening because somehow the active notebook's cells are not in the expected workspace. But I haven't been able to repro this, so I'm not positive.

When this repros in the sample repo, the user is repeatedly and rapidly switching venvs by switching tabs with the Python Envy extension installed. Apparently when this repros in a real monorepo it doesn't require so much tab switching. Maybe one is enough?

[alita-moore]

@debonte yes one is enough and I think your hypothesis makes sense. One thing I noticed is that the notebook begins to highlight if you switch to a different venv / out of that notebook. I can show you what I mean in a video if that'd help. It stops highlighting after you switch back to the notebook.

[alita-moore]

@erictraut other than the issue with it freezing it's pretty responsive. My biggest complaint is that it causes the cpu usage to spike when switching venv and sometimes if I switch quickly it takes a while to catch up. Is it possible to cache the indexing for the venv so that it requires less indexing between switches? Also, if I'm not mistaken pylance seems to index fully for each venv change even if I switch before it finishes (e.g. say I switch from venv A to venv B and then back to A before B finishes indexing, it seems that pylance will finish indexing B despite this switch), but I may be mistaken there.

[debonte]

One thing I noticed is that the notebook begins to highlight if you switch to a different venv / out of that notebook. I can show you what I mean in a video if that'd help

Yeah, a video of that would be interesting.

Here's an experiment that I'd be curious about:

1. Open two notebooks in the same project (same venv).
2. Repro the issue in one of those two notebooks.
3. If you switch between those two notebook tabs now, does one of them have functional semantic highlighting while the other one does not?
4. If so, if you close the notebook that is not showing semantic highlighting (which hopefully makes the sibling notebook in the same venv the active tab, because we want to avoid switching venvs), and then reopen that notebook, does semantic highlighting work in that notebook now?

[alita-moore]

@debonte I was having trouble reproducing the issue, but I might have found something. It seems that if I import tqdm it becomes a lot easier to reproduce the issue. I have updated the linked codesandbox with tqdm to make trying it easier for you. For reference here's a video of me reproducing it: https://www.loom.com/share/7c2ad9054cf042a0990c6f20118820a1

[debonte]

It seems that if I import tqdm it becomes a lot easier to reproduce the issue.

Good clue! I'm out of town this week, so i won't be able to look at this until next week.

[alita-moore]

I've noticed switching sometimes stops highlighting regular python files as well as jupyter notebooks.

[alita-moore]

Restart Lang

@debonte it seems that this isn't available in jupyter notebook for some reason, i.e. in my vscode if a jupyter notebook is focused it doesn't show up, but it does if I have a python file selected. (I'm referring to the "restart language server" that we had talked about previously)

[alita-moore]

Also this may be a separate issue, but it seems that when switching environments and trying to check for a funciton's references while indexing it sometimes kills the language server. The log outputs are as follows.

2024-07-03 06:10:11.188 [info] Auto-Indent enabled
2024-07-03 06:10:18.051 [info] 
<--- Last few GCs --->

[15379:0x6f696b0]  5120603 ms: Mark-Compact 7837.9 (8227.3) -> 7826.9 (8230.7) MB, 1900.86 / 0.00 ms  (average mu = 0.362, current mu = 0.459) allocation failure; scavenge might not succeed
[15379:0x6f696b0]  5123788 ms: Mark-Compact 7842.8 (8230.7) -> 7830.7 (8232.2) MB, 3136.87 / 0.00 ms  (average mu = 0.179, current mu = 0.015) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----


2024-07-03 06:10:18.074 [info]  1: 0xb80c78 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/bin/node]

2024-07-03 06:10:18.075 [info]  2: 0xeee1b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/bin/node]

2024-07-03 06:10:18.075 [info]  3: 0xeee497 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/bin/node]

2024-07-03 06:10:18.076 [info]  4: 0x1100035  [/usr/bin/node]

2024-07-03 06:10:18.076 [info]  5: 0x11005c4 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/bin/node]

2024-07-03 06:10:18.077 [info]  6: 0x11174b4 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/usr/bin/node]

2024-07-03 06:10:18.077 [info]  7: 0x1117ccc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]

2024-07-03 06:10:18.078 [info]  8: 0x10edfd1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]

2024-07-03 06:10:18.078 [info]  9: 0x10ef165 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]

2024-07-03 06:10:18.079 [info] 10: 0x10cc7b6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]

2024-07-03 06:10:18.079 [info] 11: 0x15285e6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
12: 0x70cfc3ed9ef6 

2024-07-03 06:11:47.548 [info] [Error - 6:11:47 AM] Stopping server timed out

Sorry, this might be a different issue

I think aside from reducing the weight of changing the env by caching indexes or something along those lines, it would be nice also if the server would restart in the event of a failure like the above. I currently have to reload vscode because it doesn't seem to recover from the heap out of memory error. note that doing "restart language server" does not seem to fix the problem, only a reload works.

[debonte]

Thanks for your continued help on this. I'm now able to reproduce the issue on codesandbox and have confirmed that the problem is that we, for some reason, removed the target notebook from our internal view of the project. I'll continue investigating and let you know when I figure it out.

Please open a separate issue for the crash.

[heejaechang]

This issue has been fixed in prerelease version 2024.7.103, which we've just released. You can find the changelog here: CHANGELOG.md

[alita-moore]

@heejaechang it seems that it did kind of fix the problem but it still happens if you switch enough and in addition it now affects standard python files as well as notebooks :(

EDIT: it might be that my extension reverted to non-pre-release. I'll update if it still breaks.

Edit 2: it seems that this issue is fixed but there are still other issues, I'll create a new issue for those. Thank you guys so much!