Final annotation is not recognized across notebook cells

[GF-Huang]

Environment data

• Pylance version: v2024.9.2
• OS and version: Win11 23H2
• Python version (& distribution if applicable, e.g. Anaconda): Conda Python 3.11.9

Code Snippet

from typing import Final

test: Final[int] = 0
test = 1

Repro Steps

1. XXX

Expected behavior

Show me some warning like: "you can't change constant".

Actual behavior

No warning.

Logs

XXX

[rchiodo]

Thanks for the issue. You have to set your typechecking mode higher than 'off' for an error to show up:

This is what I get with typeCheckingMode set to basic

[GF-Huang]

Thanks for the issue. You have to set your typechecking mode higher than 'off' for an error to show up:

This is what I get with typeCheckingMode set to basic

Is it not works for jupyter notebook? I had set to basic but not work, same as previous.

[rchiodo]

It works for me in a jupyter notebook. Is this an untitled notebook? The setting applies to things in the workspace, it might be the untitled notebook is not considered part of the same settings.

[rchiodo]

It does work for me in the untitled case too though. Must be the typecheckingmode is not being applied for some reason to your notebook.

[GF-Huang]

Not work. I had tried both untitled or saved file.

[rchiodo]

Can you include your logs after running 'Pylance : Start Logging'. It should tell us which workspace we think the notebook is in.

[GF-Huang]

Hope these helpful.

pylance_2024.9.2_id_3.txt

1.mp4

[rchiodo]

Hmm, I can reproduce it now too using workspace settings.json. Not sure why that matters.

[rchiodo]

Yeah there's an error reading the settings for some reason. Here:

2024-09-25 11:38:05.018 [info] [Error - 11:38:05 AM] (17092) Error reading settings: TypeError: Cannot set properties of undefined (setting 'strictListInference')
2024-09-25 11:38:05.019 [info] (17092) Notebook settings returned for workspace: file:///c%3A/Users/rchiodo/source/testing/notebooks: {
  "autoSearchPaths": false,
  "openFilesOnly": true,
  "watchForConfigChanges": false,
  "watchForSourceChanges": false,
  "watchForLibraryChanges": false,
  "typeCheckingMode": "off", <-- This is coming out wrong
  "diagnosticSeverityOverrides": {
    "reportMissingImports": "none",
    "reportMissingModuleSource": "none"
  },
  "diagnosticBooleanOverrides": {
    "enableReachabilityAnalysis": false
  },
  "enableExtractCodeAction": false,
  "callArgumentNameInlayHints": "off",
  "variableInlayTypeHints": false,
  "pytestParametersInlayTypeHints": false,
  "functionReturnInlayTypeHints": false,
  "aiCodeActions": {}
}

[rchiodo]

Figured out the problem internally. We're not propagating the imports from the first cell to the second cell.

If you put the import in the same cell, the error shows up:

from typing import Final

x: Final[int] = 3
x = 212

[rchiodo]

I can reproduce this with two python files (no need for a notebook). Should be a pyright issue?

Create two files:

exporting_file.py:

from typing import Final

importing_file.py

from exporting_file import *

x: Final[int] = 1
x = 2 # should be an error here and there isn't

This is basically the internal structure of how notebook cells are linked together.

[erictraut]

@rchiodo, you can't re-export Final or ClassVar and alias them from another file. They must be imported directly from typing to work in pyright. This is a requirement of the binder, which needs to understand Final before type evaluation is possible. If you add from typing import Final to importing_file.py, it will work as you expect.

In most languages, Final and ClassVar would be keywords rather than regular identifiers, but unfortunately Python doesn't have such keywords, so we need to resort to some hacks in pyright's binder to handle these.

[rchiodo]

Hmm, that means in a notebook they have to be in the same cell then as our workaround for cells referencing each other doesn't work in this case.

[rchiodo]

Transferring back to Pylance. We might have to special case this for notebooks then.

[rchiodo]

@erictraut is it only TypeAliases that can't be reexported? Or might there be other cases? I mean functions and variables obviously work and normal imports as well.

Like this is fine:

# Exporting_file.py
import pytest as py

# Importing_file.py
from exporting_file import *

py.foo()

[erictraut]

It's only the typing.Final and typing.ClassVar special forms (and any local aliases to these, such as _FinalAlias in the case of from typing import Final as _FinalAlias). [This list used to also include typing.Literal and typing.TypeAlias, but I found ways to defer the evaluation of these. Deferring evaluation of Final and ClassVar is much more problematic.]

How is pylance currently making symbols from one cell visible to subsequent cells? Are you synthesizing a wildcard import statement for pyright, like you show in the above sample?

[rchiodo]

How is pylance currently making symbols from one cell visible to subsequent cells? Are you synthesizing a wildcard import statement for pyright, like you show in the above sample?

Sort of? The binder binds them as implicit imports, but I'm having trouble finding how that affects the import resolver. Meaning how they are actually considered. Maybe we can tweak something in the import resolver for this special case.

[rchiodo]

Oh it's the binding that's linking everything together. It's not the same as a wildcard import.

This line here:
https://github.com/microsoft/pyright/blob/a209885f0d612dfecbf6c624d162537cb7734f12/packages/pyright-internal/src/analyzer/program.ts#L1772

It specifies the previous cell as the list of builtins. This makes all of the symbols in the previous cell count as builtin symbols for the next cell.

[erictraut]

OK, got it. I have some ideas for how to fix this, but they're going to require some explorations. I'll get back to you.

[erictraut]

I've had a chance to explore some options.

The underlying issue here is that the binder needs to make some assumptions about Final because it can't evaluate types (since it runs before the type evaluator). It looks specifically for an import of Final from typing or typing_extension or a wildcard import from one of these two modules. This assumption is somewhat fragile but generally works. The pylance code that handles "chained files" breaks this assumption because it can introduce a Final symbol (or an alias thereof) into the namespace without the binder having any knowledge of it.

I explored a couple of options:

1. Remove the fragility in the binder by pushing all evaluation of Final to the type evaluation phase. This is not only a very heavy lift, involving many code changes inside of pyright and pylance, but it also has a significant performance implications for language server features — and especially the symbol indexer in pylance. That's because the type evaluator would need to be consulted for every symbol with a variable declaration to determine whether it is final or not — and therefore whether it should be displayed as a "constant" or a "variable" in hover text and completion suggestions. If we dropped the feature that displays "constants" and "variables" differently, it would mitigate the performance impact somewhat, but I don't think we want to lose that feature.
2. Plug the specific hole that breaks the assumption in the binder. This doesn't fully address the fragility, but it does allow the "chained files" mechanism to propagate information from the chained builtins module and any typing module imports (like Final) that were encountered when it was being bound.

Option 2 is more surgical, so I went with that solution. This will be included in the next release of pyright. @rchiodo, when you pull the code from pyright, please verify that it works with pylance's notebook support. You may want to add a pylance test case specifically for this.

[rchiodo]

Option 2 is more surgical, so I went with that solution. This will be included in the next release of pyright. @rchiodo, when you pull the code from pyright, please verify that it works with pylance's notebook support. You may want to add a pylance test case specifically for this.

Thanks a lot. Will do.

[rchiodo]

@erictraut I don't think the fix worked.

I believe this code here is assuming there's more than one assignment in the same file:
https://github.com/microsoft/pyrx/blob/15c2c0478c9a32e5507788ef7069e367fad473a8/packages/pyright/packages/pyright-internal/src/analyzer/checker.ts#L3231-L3232

The reportRedeclaration doesn't get logged because the checker is only running on the one cell so the sawAssignment always starts as false.

I'm not sure but could sawAssignment default to true if sawFinal is true? That would fix the problem, but maybe I'm misunderstanding what sawFinal means.

[erictraut]

@rchiodo, that's a different problem than the one I fixed above.

The problem I fixed previously allows the Final symbol to be imported in one cell and used in a subsequent cell. For example, if you use from typing import Final in one cell and x: Final[int] = 1 in another cell, it now works as expected.

The problem you're highlighting involves a different scenario: defining a Final symbol in one cell and then attempting to overwrite that symbol in another cell.

To understand why this is a problem, we need to consider how the "chained file" mechanism in pylance works. It models each cell as a different global module namespace and chains them together. It's similar to how the builtins.py module is implicitly chained to every module's namespace.

From the perspective of the type checker, each notebook cell is a separate module scope with its own symbol table. A variable assigned in a subsequent notebook cell results in a new symbol that shadows the same-named symbol of the earlier cells. This is also true in a non-notebook context with respect to builtins shadowing. For example, it's perfectly legitimate to define a local symbol in your module that shadows a builtins symbol, such as class list: ... or class int: ....

I think it's reasonable to expect that if builtins.pyi, __builtins__.pyi or any chained file defines a Final variable, then any attempt to shadow that variable in the local module's global namespace should be considered an error. I've added this additional check, which addresses the problem you're highlighting here.