Installation failing for Python 3.9 on Windows

[teresabehr]

I have trankit running on an old laptop with Anaconda (required me to downgrade hdf5, but it works), but I cannot get it working on my new laptop for the life of me. It's a brand-new system, so there are no old versions of anything interfering. I have tried both pip and installing from source, and neither works.

System Info:
Windows 10
Python 3.9.1
VisualStudio Build Tools C++ 2019

What I did:

• attempted to install with pip, ran into dependency error with rust.
• did some brief reading, installed visual studio build tools for C ++ so that I could install rust. Also installed rust.
• am now running into the error below:

...
Building wheels for collected packages: sentencepiece, tokenizers
  Building wheel for sentencepiece (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\users\tbehr\appdata\local\programs\python\python39\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\tbehr\\AppData\\Local\\Temp\\pip-install-rxqny9hb\\sentencepiece_327f06df01b54e21a430f9036c18b342\\setup.py'"'"'; __file__='"'"'C:\\Users\\tbehr\\AppData\\Local\\Temp\\pip-install-rxqny9hb\\sentencepiece_327f06df01b54e21a430f9036c18b342\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\tbehr\AppData\Local\Temp\pip-wheel-k4b2fb39'
       cwd: C:\Users\tbehr\AppData\Local\Temp\pip-install-rxqny9hb\sentencepiece_327f06df01b54e21a430f9036c18b342\
  Complete output (15 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.9
  copying sentencepiece.py -> build\lib.win-amd64-3.9
  running build_ext
  building '_sentencepiece' extension
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\tbehr\appdata\local\programs\python\python39\include -Ic:\users\tbehr\appdata\local\programs\python\python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29333\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /Tpsentencepiece_wrap.cxx /Fobuild\temp.win-amd64-3.9\Release\sentencepiece_wrap.obj /MT /I..\build\root\include
  cl : Command line warning D9025 : overriding '/MD' with '/MT'
  sentencepiece_wrap.cxx
  sentencepiece_wrap.cxx(2777): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for sentencepiece 

Tried again in a new, empty virtualenv and got a slightly different error, also when building sentencepiece. (Also tried cloning the repo and installing from source...got an encoding error, but I suspect that's a Microsoft/different issue, so I'm not going to worry about that right now.)

Based on this issue, I think this is because trankit forces installation of a lower version of sentencepiece (my old laptop has v0.1.91), which doesn't work for Python 3.9. v0.1.95 of sentencepiece does, and I can install that separately, but when pip-installing trankit, it uninstalls v0.1.95 and tries to install the lower version that doesn't work.

Is there a workaround for this other than using a lower version of Python? (I guess that's not a major crisis, but it's easier not to have to switch back and forth between Python 3.7 and 3.9.) Or is there a way to update the setup files to reflect the newer version of sentencepiece?

ETA: I have this working successfully in a venv with Python 3.7, so this does appear to be an issue with 3.9 only.

[minhhdvn]

Hi @teresabehr,
Thanks for letting us know.

We think the issue comes from adapter-transformers. It requires the installation of the exact version of 0.1.91 for sentencepiece, as shown here: https://github.com/Adapter-Hub/adapter-transformers/blob/master/setup.py
As Trankit is built on top of adapter-transformers, you may have to downgrade Python to the versions of 3.6, 3.7, or 3.8 to solve the problem.

[teresabehr]

@minhvannguyen
Noted. Thanks for the feedback!

Closing this issue in that case, since it looks like the update would have to be made to adapter-transformers.