Cannot install sentencepiece with Python 3.9 on Windows #591

seemethere · 2020-12-03T18:47:44Z

Currently adding Python 3.9 support for pytorch/text and ran into an issue installing sentencepiece for Python 3.9 on windows. (CircleCI logs)

  ERROR: Failed building wheel for sentencepiece
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\circleci\project\env\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-trvw9qva\\sentencepiece_6ae2202249f44bf5b7a3902ec8532c93\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-trvw9qva\\sentencepiece_6ae2202249f44bf5b7a3902ec8532c93\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\circleci\AppData\Local\Temp\pip-record-xi27zjv8\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\circleci\project\env\Include\sentencepiece'
         cwd: C:\Users\circleci\AppData\Local\Temp\pip-install-trvw9qva\sentencepiece_6ae2202249f44bf5b7a3902ec8532c93\
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.9
    creating build\lib.win-amd64-3.9\sentencepiece
    copying src\sentencepiece/__init__.py -> build\lib.win-amd64-3.9\sentencepiece
    copying src\sentencepiece/sentencepiece_model_pb2.py -> build\lib.win-amd64-3.9\sentencepiece
    copying src\sentencepiece/sentencepiece_pb2.py -> build\lib.win-amd64-3.9\sentencepiece
    running build_ext
    building 'sentencepiece._sentencepiece' extension
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    creating build\temp.win-amd64-3.9\Release\src
    creating build\temp.win-amd64-3.9\Release\src\sentencepiece
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\circleci\project\env\include -IC:\Users\circleci\project\env\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /Tpsrc/sentencepiece/sentencepiece_wrap.cxx /Fobuild\temp.win-amd64-3.9\Release\src/sentencepiece/sentencepiece_wrap.obj /MT /I..\build\root\include
    cl : Command line warning D9025 : overriding '/MD' with '/MT'
    sentencepiece_wrap.cxx
    src/sentencepiece/sentencepiece_wrap.cxx(2777): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

This is a duplicate of #452, but no real solution to building from source seems to have come from that so I have opened a new issue

Is there a workaround for getting this dependency?

cc @taku910

The text was updated successfully, but these errors were encountered:

zcain117 · 2020-12-03T19:29:28Z

@taku910 have you seen this error before?

lunixbochs · 2020-12-11T10:59:08Z

The appveyor image they're using doesn't support python 3.9
Doesn't seem like this package is trivially pip installable if it's not prebuilt on windows - seems to require the C++ sentencepiece built and installed first.

appveyor/ci#3541 (comment)

lunixbochs · 2020-12-12T08:38:12Z

You can manually grab a windows cp39 wheel from here until the maintainers cut a new release: https://ci.appveyor.com/project/taku910/sentencepiece/build/job/djp9386oghmdunxg/artifacts

ghm · 2020-12-19T13:11:40Z

I had the same problem. My system is Red Hat Enterprise Linux Server release 6.4 (Santiago). Where can I manually grab a Linux cp37 wheel with Python 3.7?

The versions of here are not suitable for my system.
When I install sentencepiece-0.1.94-cp37-cp37m-manylinux2014_x86_64.whl, as the following:

But when I import it, it was as follows:

taku910 · 2021-01-10T06:35:51Z

v0.19.5 supports Python 3.9 & windows

BastianZim · 2021-02-06T16:20:38Z

I'm running into the same problem with pip. Installing from source using setup.py build and setup.py install works but installing it using python -m pip install . -vv doesn't.
The error is src/sentencepiece/sentencepiece_wrap.cxx(2809): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory

Has anyone figured out why?

Logs: https://github.com/conda-forge/sentencepiece-feedstock/runs/1844854080

IMJONEZZ · 2021-06-08T10:14:49Z

Bumping this issue again, Python 3.8 can't install sentencepiece on Windows through pip or any available wheels.

Dontpanice · 2021-06-28T08:50:08Z

Hi,

I got the same issue but for me the problem was the following. (Using python 3.9)

installing the package from packages/sentencepiece/0.1.96/sentencepiece-0.1.96.tar.gz yielded this exact error as first post in this thread. But if I insalled it form a wheel, packages/sentencepiece/0.1.95/sentencepiece-0.1.95-cp39-cp39-win_amd64.whl
it worked.

cibernicola · 2022-09-19T10:38:16Z

Error alive again, Windows 10, Python 3.10.7

` Attempting uninstall: sentencepiece
Found existing installation: sentencepiece 0.1.97
Uninstalling sentencepiece-0.1.97:
Successfully uninstalled sentencepiece-0.1.97
Running setup.py install for sentencepiece ... error
error: subprocess-exited-with-error

× Running setup.py install for sentencepiece did not run successfully.
│ exit code: 1
╰─> [24 lines of output]
C:\Python310\lib\site-packages\setuptools\dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running install
C:\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-310
creating build\lib.win-amd64-cpython-310\sentencepiece
copying src\sentencepiece/init.py -> build\lib.win-amd64-cpython-310\sentencepiece
copying src\sentencepiece/sentencepiece_model_pb2.py -> build\lib.win-amd64-cpython-310\sentencepiece
copying src\sentencepiece/sentencepiece_pb2.py -> build\lib.win-amd64-cpython-310\sentencepiece
running build_ext
building 'sentencepiece._sentencepiece' extension
creating build\temp.win-amd64-cpython-310
creating build\temp.win-amd64-cpython-310\Release
creating build\temp.win-amd64-cpython-310\Release\src
creating build\temp.win-amd64-cpython-310\Release\src\sentencepiece
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Python310\include -IC:\Python310\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /EHsc /Tpsrc/sentencepiece/sentencepiece_wrap.cxx /Fobuild\temp.win-amd64-cpython-310\Release\src/sentencepiece/sentencepiece_wrap.obj /MT /I..\build\root\include
cl : L¡nea de comandos warning D9025 : invalidando '/MD' con '/MT'
sentencepiece_wrap.cxx
src/sentencepiece/sentencepiece_wrap.cxx(2809): fatal error C1083: No se puede abrir el archivo incluir: 'sentencepiece_processor.h': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\cl.exe' failed with exit code 2
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
Rolling back uninstall of sentencepiece
Moving to c:\python310\lib\site-packages\sentencepiece-0.1.97.dist-info
from C:\Python310\Lib\site-packages~entencepiece-0.1.97.dist-info
Moving to c:\python310\lib\site-packages\sentencepiece
from C:\Python310\Lib\site-packages~entencepiece
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure`

Edit:
This path: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64" exists, and cl.exe is there too.

Miaohy12138 · 2022-10-16T08:16:12Z

I got the same thing

FrancescoFernicola · 2022-11-21T14:25:39Z

Hello everyone,

I receive the same message with python 3.11.0 on Win 10.
I also tried installing from source but I receive the same error message:

src/sentencepiece/sentencepiece_wrap.cxx(2822): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory

Catalyst-Studio · 2023-02-24T06:49:00Z

same here

Hello everyone,

I receive the same message with python 3.11.0 on Win 10. I also tried installing from source but I receive the same error message:

src/sentencepiece/sentencepiece_wrap.cxx(2822): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory

same here

jpschw · 2023-03-20T03:53:13Z

Getting the same error

tjohnman · 2023-03-20T13:07:24Z

I had the same issue and I followed the instructions to build from source here and they worked:
https://github.com/google/sentencepiece/blob/master/python/README.md

The only difference is I was using CMake GUI and Visual Studio to generate a solution and build the INSTALL project it generated.

Juniornewxt · 2023-04-09T06:04:06Z

It worked for me using an older version of python, I used 3.10

kaleidosgu · 2023-07-27T08:14:39Z

I didn't downgrade Python, instead I modified the requirements.txt file and added the following version information:
sentencepiece==0.1.99

seemethere mentioned this issue Dec 3, 2020

[v0.8.x] Add Python 3.9 support pytorch/text#1088

Merged

taku910 closed this as completed Jan 10, 2021

teresabehr mentioned this issue Jan 28, 2021

Installation failing for Python 3.9 on Windows nlp-uoregon/trankit#4

Closed

BastianZim mentioned this issue Feb 6, 2021

Add macOS conda-forge/sentencepiece-feedstock#17

Closed

8 tasks

shirayu mentioned this issue Aug 11, 2021

Update Sentencepiece and add python 3.9 to CI espnet/espnet#3422

Merged

svlandeg mentioned this issue Oct 4, 2021

sentencepiece_wrap.cxx(2777): fatal error C1083: Impossible to open file include : 'sentencepiece_processor.h' : No such file or directory explosion/spaCy#9360

Closed

mthrok mentioned this issue Feb 17, 2022

Add Python 3.10 (build and test) pytorch/audio#2224

Closed

kbatsuren mentioned this issue Dec 1, 2022

Cannot install sentencepiece with Python 3.11 on Windows #796

Closed

pkubik mentioned this issue Mar 25, 2023

Error building wheel #827

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot install sentencepiece with Python 3.9 on Windows #591

Cannot install sentencepiece with Python 3.9 on Windows #591

seemethere commented Dec 3, 2020 •

edited

Loading

zcain117 commented Dec 3, 2020

lunixbochs commented Dec 11, 2020

lunixbochs commented Dec 12, 2020

ghm commented Dec 19, 2020 •

edited

Loading

taku910 commented Jan 10, 2021

BastianZim commented Feb 6, 2021

IMJONEZZ commented Jun 8, 2021

Dontpanice commented Jun 28, 2021

cibernicola commented Sep 19, 2022 •

edited

Loading

Miaohy12138 commented Oct 16, 2022

FrancescoFernicola commented Nov 21, 2022

Catalyst-Studio commented Feb 24, 2023

jpschw commented Mar 20, 2023

tjohnman commented Mar 20, 2023

Juniornewxt commented Apr 9, 2023

kaleidosgu commented Jul 27, 2023

Cannot install sentencepiece with Python 3.9 on Windows #591

Cannot install sentencepiece with Python 3.9 on Windows #591

Comments

seemethere commented Dec 3, 2020 • edited Loading

zcain117 commented Dec 3, 2020

lunixbochs commented Dec 11, 2020

lunixbochs commented Dec 12, 2020

ghm commented Dec 19, 2020 • edited Loading

taku910 commented Jan 10, 2021

BastianZim commented Feb 6, 2021

IMJONEZZ commented Jun 8, 2021

Dontpanice commented Jun 28, 2021

cibernicola commented Sep 19, 2022 • edited Loading

Miaohy12138 commented Oct 16, 2022

FrancescoFernicola commented Nov 21, 2022

Catalyst-Studio commented Feb 24, 2023

jpschw commented Mar 20, 2023

tjohnman commented Mar 20, 2023

Juniornewxt commented Apr 9, 2023

kaleidosgu commented Jul 27, 2023

seemethere commented Dec 3, 2020 •

edited

Loading

ghm commented Dec 19, 2020 •

edited

Loading

cibernicola commented Sep 19, 2022 •

edited

Loading