Refactor deepspeed-chat into a python package. #731
Conversation
microsoft-fevieira
requested review from
RezaYazdaniAminabadi,
ShadenSmith,
arashb,
awan-10,
conglongli,
duli2012,
eltonzheng,
jeffra,
minjiaz,
mrwyattii,
samyam,
tjruwase,
xiaoxiawu-microsoft and
yaozhewei
as code owners
Microsoft master fix merge conflicts
applications/DeepSpeed-Chat/setup.py
Outdated
| url= | ||
| "https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat", | ||
| include_package_data=True, | ||
| packages=setuptools.find_packages(include=['deepspeed_chat']), |
There was a problem hiding this comment.
Let's name the package deepspeed-chat (There is a package named deepspeed-mii) so let's follow the convention of hyphen instead of underscore.
There was a problem hiding this comment.
@jeffra and I discussed this a bit. Let me write what we decided here. PyPI name will be deepspeed-chat and the python package will be dschat since python imports won't work with hyphens
| import setuptools | ||
|
|
||
| setuptools.setup( | ||
| name="deepspeed-chat", |
There was a problem hiding this comment.
@mrwyattii - looks like we are thinking of making this into a python package, should we add scripts to publish this then?
| install_requires=[ | ||
| "datasets>=2.8.0", "sentencepiece>=0.1.97", "protobuf==3.20.3", | ||
| "accelerate>=0.15.0", "torch>=1.12.0", "deepspeed>=0.9.2", | ||
| "transformers", "tensorboard" |
There was a problem hiding this comment.
I believe our transformers dependency here is not open, looking at the original requirements.txt that is: https://github.com/microsoft/DeepSpeedExamples/blob/60e412eaa7275212e240f31055fc8b814ebe653f/applications/DeepSpeed-Chat/requirements.txt#L7
This PR updates the ds-chat workflow to work with the `deepspeed-chat` install once this PR completes in DeepSpeedExamples: deepspeedai/DeepSpeedExamples#731 Manual workflow run: https://github.com/microsoft/DeepSpeed/actions/runs/6738638809/job/18318636818
…i#4598) This PR updates the ds-chat workflow to work with the `deepspeed-chat` install once this PR completes in DeepSpeedExamples: deepspeedai/DeepSpeedExamples#731 Manual workflow run: https://github.com/microsoft/DeepSpeed/actions/runs/6738638809/job/18318636818
Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: Michael Santacroce <santacml@users.noreply.github.com> Co-authored-by: Lev Kurilenko <lekurile@microsoft.com> Co-authored-by: Lev Kurilenko <113481193+lekurile@users.noreply.github.com>
6 participants
No description provided.