Failed to install sentencepiece

[sirnails]

Hello folks,

I'm strugling to install fastchat, it seems to fail to install sentencepiece.
Any help appreciated!

C:\chat\FastChat>pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 65.5.0
tokenizers 0.13.3
wcwidth    0.2.6

C:\chat\FastChat>python --version
Python 3.11.3

This is the output when installing:

C:\chat\FastChat>pip3 install -e .
Obtaining file:///C:/chat/FastChat
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting transformers@ git+https://github.com/huggingface/transformers.git
  Cloning https://github.com/huggingface/transformers.git to c:\users\msn\appdata\local\temp\pip-install-rcqm1i7z\transformers_cd16d0a0811b4ee1b1d70cc4eb7a391d
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git 'C:\Users\msn\AppData\Local\Temp\pip-install-rcqm1i7z\transformers_cd16d0a0811b4ee1b1d70cc4eb7a391d'
  Resolved https://github.com/huggingface/transformers.git to commit 656e869a4523f6a0ce90b3aacbb05cc8fb5794bb
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting accelerate
  Using cached accelerate-0.18.0-py3-none-any.whl (215 kB)
Collecting fastapi
  Using cached fastapi-0.95.0-py3-none-any.whl (57 kB)
Collecting gradio==3.23
  Using cached gradio-3.23.0-py3-none-any.whl (15.8 MB)
Collecting markdown2[all]
  Using cached markdown2-2.4.8-py2.py3-none-any.whl (38 kB)
Collecting numpy
  Using cached numpy-1.24.2-cp311-cp311-win_amd64.whl (14.8 MB)
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting sentencepiece
  Using cached sentencepiece-0.1.97.tar.gz (524 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: tokenizers>=0.12.1 in c:\program files\python311\lib\site-packages (from fschat==0.1.9) (0.13.3)
Collecting torch
  Using cached torch-2.0.0-cp311-cp311-win_amd64.whl (172.3 MB)
Collecting uvicorn
  Using cached uvicorn-0.21.1-py3-none-any.whl (57 kB)
Collecting wandb
  Using cached wandb-0.14.2-py3-none-any.whl (2.0 MB)
Collecting prompt-toolkit>=3.0.0
  Downloading prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)
     ---------------------------------------- 385.8/385.8 kB 1.2 MB/s eta 0:00:00
Collecting rich>=10.0.0
  Using cached rich-13.3.3-py3-none-any.whl (238 kB)
Collecting aiofiles
  Using cached aiofiles-23.1.0-py3-none-any.whl (14 kB)
Collecting aiohttp
  Using cached aiohttp-3.8.4-cp311-cp311-win_amd64.whl (317 kB)
Collecting altair>=4.2.0
  Using cached altair-4.2.2-py3-none-any.whl (813 kB)
Collecting ffmpy
  Using cached ffmpy-0.3.0.tar.gz (4.8 kB)
  Preparing metadata (setup.py) ... done
Collecting fsspec
  Using cached fsspec-2023.4.0-py3-none-any.whl (153 kB)
Collecting httpx
  Using cached httpx-0.23.3-py3-none-any.whl (71 kB)
Collecting huggingface-hub
  Using cached huggingface_hub-0.13.4-py3-none-any.whl (200 kB)
Collecting jinja2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting markdown-it-py[linkify]>=2.0.0
  Using cached markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
Collecting markupsafe
  Using cached MarkupSafe-2.1.2-cp311-cp311-win_amd64.whl (16 kB)
Collecting matplotlib
  Using cached matplotlib-3.7.1-cp311-cp311-win_amd64.whl (7.6 MB)
Collecting mdit-py-plugins<=0.3.3
  Using cached mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
Collecting orjson
  Using cached orjson-3.8.9-cp311-none-win_amd64.whl (202 kB)
Collecting pandas
  Using cached pandas-2.0.0-cp311-cp311-win_amd64.whl (11.2 MB)
Collecting pillow
  Using cached Pillow-9.5.0-cp311-cp311-win_amd64.whl (2.5 MB)
Collecting pydantic
  Using cached pydantic-1.10.7-cp311-cp311-win_amd64.whl (2.1 MB)
Collecting pydub
  Using cached pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Collecting python-multipart
  Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting pyyaml
  Using cached PyYAML-6.0-cp311-cp311-win_amd64.whl (143 kB)
Collecting semantic-version
  Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting websockets>=10.0
  Using cached websockets-11.0.1-cp311-cp311-win_amd64.whl (124 kB)
Collecting wcwidth
  Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)
Collecting pygments<3.0.0,>=2.13.0
  Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-win_amd64.whl (252 kB)
Collecting filelock
  Using cached filelock-3.11.0-py3-none-any.whl (10.0 kB)
Collecting sympy
  Using cached sympy-1.11.1-py3-none-any.whl (6.5 MB)
Collecting networkx
  Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Collecting starlette<0.27.0,>=0.26.1
  Using cached starlette-0.26.1-py3-none-any.whl (66 kB)
Collecting wavedrom
  Using cached wavedrom-2.0.3.post3.tar.gz (137 kB)
  Preparing metadata (setup.py) ... done
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp311-cp311-win_amd64.whl (96 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting regex!=2019.12.17
  Using cached regex-2023.3.23-cp311-cp311-win_amd64.whl (267 kB)
Collecting tqdm>=4.27
  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting h11>=0.8
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting GitPython!=3.1.29,>=1.0.0
  Using cached GitPython-3.1.31-py3-none-any.whl (184 kB)
Collecting sentry-sdk>=1.0.0
  Using cached sentry_sdk-1.19.1-py2.py3-none-any.whl (199 kB)
Collecting docker-pycreds>=0.4.0
  Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting pathtools
  Using cached pathtools-0.1.2.tar.gz (11 kB)
  Preparing metadata (setup.py) ... done
Collecting setproctitle
  Using cached setproctitle-1.3.2-cp311-cp311-win_amd64.whl (11 kB)
Requirement already satisfied: setuptools in c:\program files\python311\lib\site-packages (from wandb->fschat==0.1.9) (65.5.0)
Collecting appdirs>=1.4.3
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting protobuf!=4.21.0,<5,>=3.19.0
  Using cached protobuf-4.22.1-cp310-abi3-win_amd64.whl (420 kB)
Collecting entrypoints
  Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting jsonschema>=3.0
  Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting toolz
  Using cached toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting colorama
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting six>=1.4.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.10-py3-none-any.whl (62 kB)
Collecting mdurl~=0.1
  Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting linkify-it-py<3,>=1
  Using cached linkify_it_py-2.0.0-py3-none-any.whl (19 kB)
Collecting python-dateutil>=2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting tzdata>=2022.1
  Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting anyio<5,>=3.4.0
  Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting attrs>=17.3.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.4-cp311-cp311-win_amd64.whl (28 kB)
Collecting async-timeout<5.0,>=4.0.0a3
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.8.2-cp311-cp311-win_amd64.whl (55 kB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.3.3-cp311-cp311-win_amd64.whl (32 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting httpcore<0.17.0,>=0.15.0
  Using cached httpcore-0.16.3-py3-none-any.whl (69 kB)
Collecting rfc3986[idna2008]<2,>=1.3
  Using cached rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting sniffio
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting contourpy>=1.0.1
  Using cached contourpy-1.0.7-cp311-cp311-win_amd64.whl (162 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.39.3-py3-none-any.whl (1.0 MB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.4-cp311-cp311-win_amd64.whl (55 kB)
Collecting pyparsing>=2.3.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting mpmath>=0.19
  Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Collecting svgwrite
  Using cached svgwrite-1.4.3-py3-none-any.whl (67 kB)
Collecting smmap<6,>=3.0.1
  Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Using cached pyrsistent-0.19.3-cp311-cp311-win_amd64.whl (62 kB)
Collecting uc-micro-py
  Using cached uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)
Building wheels for collected packages: fschat, transformers
  Building editable for fschat (pyproject.toml) ... done
  Created wheel for fschat: filename=fschat-0.1.9-0.editable-py3-none-any.whl size=11439 sha256=d6b27e424f08488b3a12e7474070e01dc31820a0f4e71cba08defad7f66fa151
  Stored in directory: C:\Users\msn\AppData\Local\Temp\pip-ephem-wheel-cache-s7ffwqci\wheels\d4\1a\90\7b409cecf8f604c048bbcdbb8ba7edea8635772646b05e33fa
  Building wheel for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6895878 sha256=74a7d2e9ec1d2e045fd74ce2fe8aa7a77540837b3452a21c9a6db66b65c3fe78
  Stored in directory: C:\Users\msn\AppData\Local\Temp\pip-ephem-wheel-cache-s7ffwqci\wheels\32\4b\78\f195c684dd3a9ed21f3b39fe8f85b48df7918581b6437be143
Successfully built fschat transformers
Installing collected packages: wcwidth, sentencepiece, rfc3986, pytz, pydub, pathtools, mpmath, ffmpy, appdirs, websockets, urllib3, uc-micro-py, tzdata, typing-extensions, toolz, sympy, svgwrite, sniffio, smmap, six, setproctitle, semantic-version, regex, pyyaml, python-multipart, pyrsistent, pyparsing, pygments, psutil, protobuf, prompt-toolkit, pillow, packaging, orjson, numpy, networkx, multidict, mdurl, markupsafe, markdown2, kiwisolver, idna, h11, fsspec, frozenlist, fonttools, filelock, entrypoints, cycler, colorama, charset-normalizer, certifi, attrs, async-timeout, aiofiles, yarl, wavedrom, tqdm, sentry-sdk, requests, python-dateutil, pydantic, markdown-it-py, linkify-it-py, jsonschema, jinja2, gitdb, docker-pycreds, contourpy, click, anyio, aiosignal, uvicorn, torch, starlette, rich, pandas, mdit-py-plugins, matplotlib, huggingface-hub, httpcore, GitPython, aiohttp, wandb, transformers, httpx, fastapi, altair, accelerate, gradio, fschat
  DEPRECATION: sentencepiece is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error

  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      running install
      C:\Program Files\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-311
      creating build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/__init__.py -> build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/_version.py -> build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/sentencepiece_model_pb2.py -> build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/sentencepiece_pb2.py -> build\lib.win-amd64-cpython-311\sentencepiece
      running build_ext
      building 'sentencepiece._sentencepiece' extension
      creating build\temp.win-amd64-cpython-311
      creating build\temp.win-amd64-cpython-311\Release
      creating build\temp.win-amd64-cpython-311\Release\src
      creating build\temp.win-amd64-cpython-311\Release\src\sentencepiece
      "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpsrc/sentencepiece/sentencepiece_wrap.cxx /Fobuild\temp.win-amd64-cpython-311\Release\src/sentencepiece/sentencepiece_wrap.obj /std:c++17 /MT /I..\build\root\include
      cl : Command line warning D9025 : overriding '/MD' with '/MT'
      sentencepiece_wrap.cxx
      src/sentencepiece/sentencepiece_wrap.cxx(2822): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
      error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

C:\chat\FastChat>

[psikosen]

I had to downgrade to python 3.9.13 via pyenv couldn't get it to work on 3.11

[ZachariahMarrero]

there is a sentencepiece wheel for python 3.10.

I was able to build sentencepiece for python 3.11 but then ran into other issues when serving the model later. So, 3.10 may be the less troublesome way to go.

[zhisbug]

Seems it is because the Python version is too high. Please try to downgrade. Meanwhile, we should specificy the right Python versions in our toml.