Test: Semantic colorization regressions #8026

rchiodo · 2021-10-25T20:02:57Z

anyOS @connor4312
anyOS @digitarald (Optional)

Complexity: 2

Semantic tokenization of notebooks has been improved (but not final). This test plan is to verify these changes don't break other scenarios (like python files)

Setup steps:

Install some version of python
Make sure you have pip installed
Install Jupyter extension from marketplace (it has insiders only versions)
Install insiders Python extension \
Install Pylance extension from marketplace
Make sure your python language server is set to 'Pylance':

    "python.languageServer": "Pylance",

Test steps:

Create a python environment using either venv or conda. Alternatively you can use an existing environment.
Open VS code in a new folder (if using venv, make sure the venv you created is located at the same root)
Create a python file like so:

import sys
import time
print(1)
time.sleep(10)


def test():
    print('test()')

def test2():
    print('test2()')


test2()
test()
time.sleep(10)

Make sure the tokenization of test2() and test() is appropriate
The import sys should also show a different color for the sys as it's not used.

The colors may take a while to appear (the language server has to be up and running first). If they don't appear, try reloading the file. Tokenization doesn't rerun for already open files when the language server starts, so you might have to close and reopen the python file.

Try making some edits to make sure tokens are updated correctly.

The text was updated successfully, but these errors were encountered:

connor4312 · 2021-10-26T16:34:30Z

Is this the appropriate tokenization of test2() and test()? The scopes seem sensible but the color is the default editor foreground, which is different from TypeScript

rchiodo · 2021-10-26T16:39:47Z

Is this the appropriate tokenization of test2() and test()? The scopes seem sensible but the color is the default editor foreground, which is different from TypeScript

No that seems incorrect. Does hover work for test2? It should show something like so:

You may also need to close and reopen the file. It seems that token requests are not resent once the language server is running (so if the file was already open when the workspace opened, it won't colorize)

Also make sure these settings are turned on:

connor4312 · 2021-10-26T16:41:10Z

The hover does work...

Both those settings are the same as yours. Same result after a window reload.

rchiodo · 2021-10-26T16:43:52Z

What's your "python.languageServer" set to? Should be "Pylance"

This is the tokens I get:

connor4312 · 2021-10-26T16:45:46Z

It is. Here's the language server logs when python.analysis.logLevel is trace, though I don't immediately see anything particularly useful. I'm on OSX:

[Info  - 9:41:54 AM] Pylance language server 2021.10.2 (pyright 1d97208c) starting
[Info  - 9:41:54 AM] Server root directory: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist
[Info  - 9:41:54 AM] No configuration file found.
[Info  - 9:41:54 AM] No pyproject.toml file found.
[Info  - 9:41:54 AM] Setting pythonPath for service "python-sem-test": "/Users/connor/Downloads/python-sem-test/.venv/bin/python"
[Warn  - 9:41:54 AM] stubPath /Users/connor/Downloads/python-sem-test/typings is not a valid directory.
[Info  - 9:41:54 AM] Assuming Python version 3.9
[Info  - 9:41:54 AM] Assuming Python platform Darwin
Search paths for /Users/connor/Downloads/python-sem-test
  /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib
  /Users/connor/Downloads/python-sem-test
  /Users/connor/Downloads/python-sem-test/typings
  /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stubs/...
  /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/bundled/stubs
  /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9
  /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload
  /Users/connor/Downloads/python-sem-test/.venv/lib/python3.9/site-packages
[Info  - 9:41:54 AM] Searching for source files
[Info  - 9:41:54 AM] Auto-excluding /Users/connor/Downloads/python-sem-test/.venv
[Info  - 9:41:54 AM] Found 1 source file
[Info  - 9:41:54 AM] Background analysis(1) root directory: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist
[Info  - 9:41:54 AM] Background analysis(1) started
Background analysis message: setConfigOptions
Background analysis message: setImportResolver
Background analysis message: ensurePartialStubPackages
Background analysis message: setTrackedFiles
Background analysis message: markAllFilesDirty
Background analysis message: setFileOpened
Background analysis message: analyze
[BG(1)] analyzing: /Users/connor/Downloads/python-sem-test/test.py ...
[BG(1)]   parsing: /Users/connor/Downloads/python-sem-test/test.py (20ms)
[BG(1)]   parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 3ms] (71ms)
[BG(1)]   binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/builtins.pyi (36ms)
[BG(1)]   binding: /Users/connor/Downloads/python-sem-test/test.py (0ms)
[BG(1)]   checking: /Users/connor/Downloads/python-sem-test/test.py ...
[BG(1)]     parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing.pyi [fs read 2ms] (30ms)
[BG(1)]     binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing.pyi (9ms)
[BG(1)]     parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 1ms] (8ms)
[BG(1)]     binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (2ms)
[BG(1)]     parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi [fs read 1ms] (3ms)
[BG(1)]     binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi (1ms)
[BG(1)]     parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/abc.pyi [fs read 1ms] (3ms)
[BG(1)]     binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/abc.pyi (1ms)
[BG(1)]     parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/sys.pyi [fs read 1ms] (7ms)
[BG(1)]     binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/sys.pyi (4ms)
[BG(1)]     parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/time.pyi [fs read 0ms] (3ms)
[BG(1)]     binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/time.pyi (1ms)
[BG(1)]   checking: /Users/connor/Downloads/python-sem-test/test.py (106ms)
[BG(1)] analyzing: /Users/connor/Downloads/python-sem-test/test.py (236ms)
Background analysis message: resumeAnalysis
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
[FG] parsing: /Users/connor/Downloads/python-sem-test/test.py (23ms)
[FG] parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 3ms] (77ms)
[FG] binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/builtins.pyi (35ms)
[FG] binding: /Users/connor/Downloads/python-sem-test/test.py (0ms)
[FG] parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing.pyi [fs read 1ms] (35ms)
[FG] binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing.pyi (10ms)
[FG] parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 1ms] (7ms)
[FG] binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (1ms)
[FG] parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi [fs read 0ms] (2ms)
[FG] binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi (1ms)
[FG] parsing: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/abc.pyi [fs read 0ms] (1ms)
[FG] binding: /Users/connor/.vscode-insiders/extensions/ms-python.vscode-pylance-2021.10.2/dist/typeshed-fallback/stdlib/abc.pyi (0ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Received updated settings

rchiodo · 2021-10-26T16:48:12Z

That should be fine. It looks like it didn't even make a token request though. Do you have any other extensions installed that might be providing a token provider?

connor4312 · 2021-10-26T16:57:08Z

I don't think so. These are my "running extensions" with the Python file open:

Prettier - Code formatter 9.0.0
Jupyter 2021.10.1001384888
Acrolinx for Visual Studio Code 1.2.0
Pylance 2021.10.2
GitHub Codespaces 1.3.2
Mocha Test Explorer 2.13.0
Python 2021.11.1382879425-dev
Git 1.0.0
GitLens — Git supercharged 11.6.1
ESLint 2.2.2
GitHub Authentication 0.0.2
Microsoft Account 0.0.1
Remote - WSL 0.58.4

rchiodo · 2021-10-26T17:03:18Z

Hmm. I have the exact same extensions. Not sure why it's not sending the request then. What version of VS code? (@alexdima made some changes for me for this, but they shouldn't affect a raw python file).

Mine is this one:
Version: 1.62.0-insider (user setup)
Commit: dc1a6699060423b8c4d2ced736ad70195378fddf
Date: 2021-10-26T08:49:04.076Z
Electron: 13.5.1
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows_NT x64 10.0.19043

connor4312 · 2021-10-26T17:04:23Z

Yep, same build.

Version: 1.62.0-insider
Commit: dc1a6699060423b8c4d2ced736ad70195378fddf
Date: 2021-10-26T08:47:07.881Z
Electron: 13.5.1
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin x64 20.6.0

Let me know if there's more info I can get. I'm also happy to jump on a call or install a custom extension build with additional logging.

digitarald · 2021-10-26T23:30:17Z

I also hit https://github.com/microsoft/vscode/issues/135892, but with Window Reload.

rchiodo added the bug Issue identified by VS Code Team member as probable bug label Oct 25, 2021

github-actions bot added the needs-triage label Oct 25, 2021

rchiodo added testplan-item and removed bug Issue identified by VS Code Team member as probable bug needs-triage labels Oct 25, 2021

rchiodo changed the title ~~Test: Semantic colorization of notebooks (preliminary)~~ Test: Semantic colorization regressions Oct 25, 2021

ghost assigned connor4312 and digitarald Oct 26, 2021

connor4312 removed their assignment Oct 26, 2021

digitarald removed their assignment Oct 26, 2021

digitarald closed this as completed Oct 26, 2021

connor4312 mentioned this issue Oct 29, 2021

Semantic highlighting does not work if registered after file is open microsoft/vscode-python#17878

Closed

github-actions bot locked as resolved and limited conversation to collaborators Oct 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test: Semantic colorization regressions #8026

Test: Semantic colorization regressions #8026

rchiodo commented Oct 25, 2021 •

edited by digitarald

Loading

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021 •

edited

Loading

digitarald commented Oct 26, 2021

Test: Semantic colorization regressions #8026

Test: Semantic colorization regressions #8026

Comments

rchiodo commented Oct 25, 2021 • edited by digitarald Loading

Setup steps:

Test steps:

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021

rchiodo commented Oct 26, 2021

connor4312 commented Oct 26, 2021 • edited Loading

digitarald commented Oct 26, 2021

rchiodo commented Oct 25, 2021 •

edited by digitarald

Loading

connor4312 commented Oct 26, 2021 •

edited

Loading