typeshed not being found on windows user install

[Summertime]

PS D:\testing> git clone https://github.com/python/mypy.git
# no errors

PS D:\testing> cd .\mypy\

PS D:\testing\mypy> git submodule update --init typeshed
# worked, checked out '55dbb967ad006d5835590c53ab023774dc5a1341'

PS D:\testing\mypy> pip install --user --upgrade .
Processing D:\testing\mypy
Requirement already up-to-date: typed-ast<1.2.0,>=1.1.0 in c:\users\[redacted!]\appdata\roaming\python\python36\site-packages (from mypy===0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca)
Installing collected packages: mypy
  Running setup.py install for mypy ... done
Successfully installed mypy-0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca

PS D:\testing\mypy> mypy setup.py  # to test
Could not resolve typeshed subdirectories. If you are using MyPyfrom source, you need to run "git submodule --init update".Otherwise your MyPy install is broken.

The data directory (after searching for typeshed) appears to be %AppData%\Python\lib\mypy

full path: C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy

PS D:\testing\mypy> ls $Env:AppData\Python\lib\mypy

typeshed
xml

using this directory leads to more expected results

PS D:\testing\mypy> mypy --custom-typeshed-dir $Env:AppData\Python\lib\mypy\typeshed setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable

[gvanrossum]

It would seem that mypy's search for typeshed needs to be augmented with a case for this type of configuration. Mypy itself is at $HOME\appdata\roaming\python\python36\site-packages\mypy and typeshed is at $HOME\AppData\Python\lib\mypy\typeshed. Could you take a debugger and step through the default_data_dir() function (or insert some print statements) to find out where it goes wrong? I'm guessing the reason might be that it's looking for a component of the path named lib, but in your case there isn't one.

How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

[emmatyping]

I can debug this.

How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

For me the default 3.x install directory for the newer installers (3.5+) point to %USERPROFILE%\AppData\Local\Programs\Python\PythonXY instead of C:\PythonXY.

[Summertime]

sorry for a slightly-too-late response, the fix doesn't seem to work for me :(

https://gist.github.com/Summertime/403ce90d067253669796326212becce0 probably-overkill install logs are here for the exact paths of everything (installing from the latest merge)

I can give a shot at trying to work out the issue in a couple days, however the problem might be obvious from the install logs already, I'm not too sure.

[emmatyping]

Hm okay. Based on the output of the log it seems that you are using a custom compiled Python. Is that true? Sorry that fix didn't work for you.

[Summertime]

no custom compiling! I reinstalled python just to make sure, clicked the main 3.x.x download link on python.org, did the following, then ran pip install . --user in the same folder as before, same result!

[emmatyping]

Could you tell me what where mypy gives you now that you re-installed Python? Thanks!

[gvanrossum]

I wonder if (at least on Windows) the search code should apply os.path.normpath() before comparing?

…

On Sun, Sep 24, 2017 at 3:13 PM, Ethan Smith ***@***.***> wrote: Could you tell me what where mypy gives you now that you re-installed Python? Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3988 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMkCtUmPLkNqTdu6tANJwr6ik7HrBks5sltP0gaJpZM4PheqA> .

-- --Guido van Rossum (python.org/~guido)

[emmatyping]

Could you point out the comparison that you are thinking of? The part of the code in question is independent of the OS path separator to my knowledge. I'm thinking of here in particular

mypy/mypy/build.py

Line 231 in 6c409b4

if os.path.basename(lib) in ('lib', 'lib32', 'lib64') \

[gvanrossum]

I wasn't thinking of os.sep but of different cases for lib or python (on the next line). In particular the OP has Python (%AppData%\Python\lib\mypy and C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy).

[emmatyping]

Confusingly, %APPDATA% expands to %USERPROFILE%\AppData\Roaming or C:\Users\<username>\AppData\Roaming, so essentially the two are the same.

[gvanrossum]

I'm working on the hypothesis that ...\appdata\roaming\python\python36\site-packages (as quoted by OP) might be using capitalization everywhere, i.e. ...\AppData\Roaming\Python\Python36\site-packages. The only time this path is quoted by the OP it's clearly already lowercased by pip, so we have no hard evidence against this hypothesis. And if that's true, the algorithm (which seems to be searching for the python36 segment of the path) would fail since it's doing a case-sensitive comparison.

I have no idea what installer was used, so I don't want to exclude this hypothesis yet -- it seems to be using capitalization in earlier segments of the path (at least ...\AppData\Roaming\Python are quoted by the OP). Also, I found this about AppData\Roaming: https://askleo.com/whats-the-appdata-roaming-folder/

[Summertime]

path of mypy after reinstall

PS D:\[username]\testing\mypy> where.exe mypy
C:\Users\[username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

I did make sure to wipe all site-package/script directories before reinstalling too, so not an artifact of the previous install

paths and capitalization

As reported by powershell/windows (and as a result, probably the "official" capitalization that would show up:

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path .\mypy.exe | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path .\mypy.exe | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path ..\site-packages\mypy | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\Python36\site-packages\mypy

PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path ..\..\lib\mypy\typeshed | %{$_.Path}
C:\Users\[Username]\AppData\Roaming\Python\lib\mypy\typeshed

custom Installer

I'm working on seeing if this reproduces in a fresh windows 10 VM using only
https://www.python.org/ftp/python/3.6.2/python-3.6.2.exe and
https://github.com/git-for-windows/git/releases/download/v2.14.1.windows.1/Git-2.14.1-64-bit.exe

[emmatyping]

Aha, I see an issue. In the screenshots of your installer, you are using Python 3.6 32 bits. It is being installed in %USERPROFILE%\AppData\Local\.... It seems that the mypy.exe is left over (it is installed in %USERPROFILE%\AppData\Roaming).

However, I agree that the check for prefixes with python should probably be case insensitive. So I will make a PR about that.

[ilevkivskyi]

By the way #3991 broke installing mypy on Linux Mint. So if #3991 doesn't really solve this problem then I propose to remove the or os.path.basename(lib).startswith('python') line.

[emmatyping]

Hm, that is concerning. Could you describe how it is broken?

[ilevkivskyi]

...or make this check dependent on sys.platform == 'linux'.

[ilevkivskyi]

I see this when I install from git master:

tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)

[ilevkivskyi]

FWIW, removing the line I mentioned fixes the problem. But I have no idea why.

[Summertime]

it goes into appdata/roaming when done with --user, if I did it without --user it would end up in the appdata/local spot

I did say I wiped the folder!

And it reproduces in a VM too!

# after installing python like I did in the screenshots before:
# Also using absolute paths since too lazy to update my path for pip's user install location
PS [blah]\mypy> C:\Users\[differentusername!]\AppData\Roaming\Python\Python36\Scripts\mypy.exe setup.py
Could not resolve typeshed subdirectories. If you are using mypy
from source, you need to run "git submodule --init update".
Otherwise your mypy install is broken.
Python executable is located at c:\users\[differentusername!]\appdata\local\programs\python\python36-32\python.exe.
Mypy located at C:\Users\[differentusername!]\AppData\Roaming\Python\Python36\site-packages

However I will note, running the module directly works:

PS [blah]\mypy> python -m mypy .\setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable

(but for some reason my main install doesn't mirror that, dunno!) (I guess I'll consider my main install damaged for now since it repros differently)

[ilevkivskyi]

It looks like installation on Mac is also broken now. I see:

tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)

[gvanrossum]

Should we just roll back whatever broke things recently? On Sep 25, 2017 6:58 AM, "Ivan Levkivskyi" <notifications@github.com> wrote: It looks like installation on Mac is also broken now. I see: tst22.py: error: Name '__builtins__' is not defined tst22.py:1: error: No library stub file for standard library module 'typing' tst22.py:1: note: (Stub files are from https://github.com/python/typeshed) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3988 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMpNq95PXa1706JJE3P_zHYh7zUUKks5sl7E7gaJpZM4PheqA> .

[emmatyping]

I made a PR which should fix everything in #4005.