Add support for datasets in local json files#597
Add support for datasets in local json files#597conglongli merged 16 commits intodeepspeedai:masterfrom
Conversation
LeetJoe
requested review from
RezaYazdaniAminabadi,
ShadenSmith,
arashb,
awan-10,
cli99,
conglongli,
duli2012,
eltonzheng,
jeffra,
minjiaz,
mrwyattii,
samyam,
tjruwase,
xiaoxiawu-microsoft and
yaozhewei
as code owners
|
@microsoft-github-policy-service agree |
applications/DeepSpeed-Chat/training/utils/data/raw_datasets.py
Outdated
Show resolved
Hide resolved
|
Hi @conglongli thank you for your reviewing and you really offer me some good advices. |
|
@LeetJoe Most of my comments are resolved, but there are two new issues. Also you missed my ask of fixing the formatting issue. If formatting is not fixed, we can't merge it |
fix some about format
|
@conglongli I have fixed the problems you mentioned above. Thanks for your patience. |
|
@LeetJoe formatting is still failling, please do exactly the following: go to the repo |
reformat code using yapf
|
@conglongli I check the failure and find there is a tool called yapf and I have used it reformat my code. |
|
@LeetJoe yapf is just one of the checks we did. This time it was the only format issue so that I can merge. In future I highly recommend you install and use pre-commit. |
|
I have followed the code you provided above, pre-commit is installed now. |
* fix gitignore * add local dataset dir * add ignore * add local dataset support * add some about local dataset in README.md * fix some * add some in README * remove data dir * add line to gitignore * fix some following good advices * fix some about format * reformat code using yapf --------- Co-authored-by: 宋超 <sc@chainshome.org>
2 participants
README is also updated.