Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failing to install FastChat - missing sentencepiece_processor.h #847

Closed
sirnails opened this issue Apr 9, 2023 · 1 comment
Closed

failing to install FastChat - missing sentencepiece_processor.h #847

sirnails opened this issue Apr 9, 2023 · 1 comment
Labels
duplicate

Comments

@sirnails
Copy link

sirnails commented Apr 9, 2023

Hello folks,

I'm strugling to install fastchat, it seems to fail to install sentencepiece.
Any help appreciated!

C:\chat\FastChat>pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 65.5.0
tokenizers 0.13.3
wcwidth    0.2.6

C:\chat\FastChat>python --version
Python 3.11.3

This is the output when installing:

C:\chat\FastChat>pip3 install -e .
Obtaining file:///C:/chat/FastChat
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting transformers@ git+https://github.com/huggingface/transformers.git
  Cloning https://github.com/huggingface/transformers.git to c:\users\msn\appdata\local\temp\pip-install-rcqm1i7z\transformers_cd16d0a0811b4ee1b1d70cc4eb7a391d
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git 'C:\Users\msn\AppData\Local\Temp\pip-install-rcqm1i7z\transformers_cd16d0a0811b4ee1b1d70cc4eb7a391d'
  Resolved https://github.com/huggingface/transformers.git to commit 656e869a4523f6a0ce90b3aacbb05cc8fb5794bb
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting accelerate
  Using cached accelerate-0.18.0-py3-none-any.whl (215 kB)
Collecting fastapi
  Using cached fastapi-0.95.0-py3-none-any.whl (57 kB)
Collecting gradio==3.23
  Using cached gradio-3.23.0-py3-none-any.whl (15.8 MB)
Collecting markdown2[all]
  Using cached markdown2-2.4.8-py2.py3-none-any.whl (38 kB)
Collecting numpy
  Using cached numpy-1.24.2-cp311-cp311-win_amd64.whl (14.8 MB)
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting sentencepiece
  Using cached sentencepiece-0.1.97.tar.gz (524 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: tokenizers>=0.12.1 in c:\program files\python311\lib\site-packages (from fschat==0.1.9) (0.13.3)
Collecting torch
  Using cached torch-2.0.0-cp311-cp311-win_amd64.whl (172.3 MB)
Collecting uvicorn
  Using cached uvicorn-0.21.1-py3-none-any.whl (57 kB)
Collecting wandb
  Using cached wandb-0.14.2-py3-none-any.whl (2.0 MB)
Collecting prompt-toolkit>=3.0.0
  Downloading prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)
     ---------------------------------------- 385.8/385.8 kB 1.2 MB/s eta 0:00:00
Collecting rich>=10.0.0
  Using cached rich-13.3.3-py3-none-any.whl (238 kB)
Collecting aiofiles
  Using cached aiofiles-23.1.0-py3-none-any.whl (14 kB)
Collecting aiohttp
  Using cached aiohttp-3.8.4-cp311-cp311-win_amd64.whl (317 kB)
Collecting altair>=4.2.0
  Using cached altair-4.2.2-py3-none-any.whl (813 kB)
Collecting ffmpy
  Using cached ffmpy-0.3.0.tar.gz (4.8 kB)
  Preparing metadata (setup.py) ... done
Collecting fsspec
  Using cached fsspec-2023.4.0-py3-none-any.whl (153 kB)
Collecting httpx
  Using cached httpx-0.23.3-py3-none-any.whl (71 kB)
Collecting huggingface-hub
  Using cached huggingface_hub-0.13.4-py3-none-any.whl (200 kB)
Collecting jinja2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting markdown-it-py[linkify]>=2.0.0
  Using cached markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
Collecting markupsafe
  Using cached MarkupSafe-2.1.2-cp311-cp311-win_amd64.whl (16 kB)
Collecting matplotlib
  Using cached matplotlib-3.7.1-cp311-cp311-win_amd64.whl (7.6 MB)
Collecting mdit-py-plugins<=0.3.3
  Using cached mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
Collecting orjson
  Using cached orjson-3.8.9-cp311-none-win_amd64.whl (202 kB)
Collecting pandas
  Using cached pandas-2.0.0-cp311-cp311-win_amd64.whl (11.2 MB)
Collecting pillow
  Using cached Pillow-9.5.0-cp311-cp311-win_amd64.whl (2.5 MB)
Collecting pydantic
  Using cached pydantic-1.10.7-cp311-cp311-win_amd64.whl (2.1 MB)
Collecting pydub
  Using cached pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Collecting python-multipart
  Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting pyyaml
  Using cached PyYAML-6.0-cp311-cp311-win_amd64.whl (143 kB)
Collecting semantic-version
  Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting websockets>=10.0
  Using cached websockets-11.0.1-cp311-cp311-win_amd64.whl (124 kB)
Collecting wcwidth
  Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)
Collecting pygments<3.0.0,>=2.13.0
  Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-win_amd64.whl (252 kB)
Collecting filelock
  Using cached filelock-3.11.0-py3-none-any.whl (10.0 kB)
Collecting sympy
  Using cached sympy-1.11.1-py3-none-any.whl (6.5 MB)
Collecting networkx
  Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Collecting starlette<0.27.0,>=0.26.1
  Using cached starlette-0.26.1-py3-none-any.whl (66 kB)
Collecting wavedrom
  Using cached wavedrom-2.0.3.post3.tar.gz (137 kB)
  Preparing metadata (setup.py) ... done
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp311-cp311-win_amd64.whl (96 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting regex!=2019.12.17
  Using cached regex-2023.3.23-cp311-cp311-win_amd64.whl (267 kB)
Collecting tqdm>=4.27
  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting h11>=0.8
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting GitPython!=3.1.29,>=1.0.0
  Using cached GitPython-3.1.31-py3-none-any.whl (184 kB)
Collecting sentry-sdk>=1.0.0
  Using cached sentry_sdk-1.19.1-py2.py3-none-any.whl (199 kB)
Collecting docker-pycreds>=0.4.0
  Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting pathtools
  Using cached pathtools-0.1.2.tar.gz (11 kB)
  Preparing metadata (setup.py) ... done
Collecting setproctitle
  Using cached setproctitle-1.3.2-cp311-cp311-win_amd64.whl (11 kB)
Requirement already satisfied: setuptools in c:\program files\python311\lib\site-packages (from wandb->fschat==0.1.9) (65.5.0)
Collecting appdirs>=1.4.3
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting protobuf!=4.21.0,<5,>=3.19.0
  Using cached protobuf-4.22.1-cp310-abi3-win_amd64.whl (420 kB)
Collecting entrypoints
  Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting jsonschema>=3.0
  Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting toolz
  Using cached toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting colorama
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting six>=1.4.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.10-py3-none-any.whl (62 kB)
Collecting mdurl~=0.1
  Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting linkify-it-py<3,>=1
  Using cached linkify_it_py-2.0.0-py3-none-any.whl (19 kB)
Collecting python-dateutil>=2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting tzdata>=2022.1
  Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting anyio<5,>=3.4.0
  Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting attrs>=17.3.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.4-cp311-cp311-win_amd64.whl (28 kB)
Collecting async-timeout<5.0,>=4.0.0a3
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.8.2-cp311-cp311-win_amd64.whl (55 kB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.3.3-cp311-cp311-win_amd64.whl (32 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting httpcore<0.17.0,>=0.15.0
  Using cached httpcore-0.16.3-py3-none-any.whl (69 kB)
Collecting rfc3986[idna2008]<2,>=1.3
  Using cached rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting sniffio
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting contourpy>=1.0.1
  Using cached contourpy-1.0.7-cp311-cp311-win_amd64.whl (162 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.39.3-py3-none-any.whl (1.0 MB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.4-cp311-cp311-win_amd64.whl (55 kB)
Collecting pyparsing>=2.3.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting mpmath>=0.19
  Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Collecting svgwrite
  Using cached svgwrite-1.4.3-py3-none-any.whl (67 kB)
Collecting smmap<6,>=3.0.1
  Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Using cached pyrsistent-0.19.3-cp311-cp311-win_amd64.whl (62 kB)
Collecting uc-micro-py
  Using cached uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)
Building wheels for collected packages: fschat, transformers
  Building editable for fschat (pyproject.toml) ... done
  Created wheel for fschat: filename=fschat-0.1.9-0.editable-py3-none-any.whl size=11439 sha256=d6b27e424f08488b3a12e7474070e01dc31820a0f4e71cba08defad7f66fa151
  Stored in directory: C:\Users\msn\AppData\Local\Temp\pip-ephem-wheel-cache-s7ffwqci\wheels\d4\1a\90\7b409cecf8f604c048bbcdbb8ba7edea8635772646b05e33fa
  Building wheel for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6895878 sha256=74a7d2e9ec1d2e045fd74ce2fe8aa7a77540837b3452a21c9a6db66b65c3fe78
  Stored in directory: C:\Users\msn\AppData\Local\Temp\pip-ephem-wheel-cache-s7ffwqci\wheels\32\4b\78\f195c684dd3a9ed21f3b39fe8f85b48df7918581b6437be143
Successfully built fschat transformers
Installing collected packages: wcwidth, sentencepiece, rfc3986, pytz, pydub, pathtools, mpmath, ffmpy, appdirs, websockets, urllib3, uc-micro-py, tzdata, typing-extensions, toolz, sympy, svgwrite, sniffio, smmap, six, setproctitle, semantic-version, regex, pyyaml, python-multipart, pyrsistent, pyparsing, pygments, psutil, protobuf, prompt-toolkit, pillow, packaging, orjson, numpy, networkx, multidict, mdurl, markupsafe, markdown2, kiwisolver, idna, h11, fsspec, frozenlist, fonttools, filelock, entrypoints, cycler, colorama, charset-normalizer, certifi, attrs, async-timeout, aiofiles, yarl, wavedrom, tqdm, sentry-sdk, requests, python-dateutil, pydantic, markdown-it-py, linkify-it-py, jsonschema, jinja2, gitdb, docker-pycreds, contourpy, click, anyio, aiosignal, uvicorn, torch, starlette, rich, pandas, mdit-py-plugins, matplotlib, huggingface-hub, httpcore, GitPython, aiohttp, wandb, transformers, httpx, fastapi, altair, accelerate, gradio, fschat
  DEPRECATION: sentencepiece is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error

  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      running install
      C:\Program Files\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-311
      creating build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/__init__.py -> build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/_version.py -> build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/sentencepiece_model_pb2.py -> build\lib.win-amd64-cpython-311\sentencepiece
      copying src\sentencepiece/sentencepiece_pb2.py -> build\lib.win-amd64-cpython-311\sentencepiece
      running build_ext
      building 'sentencepiece._sentencepiece' extension
      creating build\temp.win-amd64-cpython-311
      creating build\temp.win-amd64-cpython-311\Release
      creating build\temp.win-amd64-cpython-311\Release\src
      creating build\temp.win-amd64-cpython-311\Release\src\sentencepiece
      "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpsrc/sentencepiece/sentencepiece_wrap.cxx /Fobuild\temp.win-amd64-cpython-311\Release\src/sentencepiece/sentencepiece_wrap.obj /std:c++17 /MT /I..\build\root\include
      cl : Command line warning D9025 : overriding '/MD' with '/MT'
      sentencepiece_wrap.cxx
      src/sentencepiece/sentencepiece_wrap.cxx(2822): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
      error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

C:\chat\FastChat>

Originally posted by @sirnails in lm-sys/FastChat#324
@sirnails sirnails changed the title Hello folks, failing to install sentencepiece Apr 9, 2023
@sirnails sirnails changed the title failing to install sentencepiece failing to install FastChat - missing sentencepiece_processor.h Apr 9, 2023
@taku910
Copy link
Collaborator

taku910 commented Apr 10, 2023

Please check #827

@taku910 taku910 closed this as completed Apr 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@taku910 @sirnails