Reproducible checkpoint for npu #27208

statelesshz · 2023-11-01T14:00:23Z

What does this PR do?

Save npu's RNG states when saving a checkpoint and set after all the data skip phase when resuming training.
Links to #11582

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

data skip phase when resuming training.

amyeroberts

Thanks for adding!

HuggingFaceDocBuilderDev · 2023-11-01T14:20:56Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

muellerzr

Thanks!

statelesshz · 2023-11-02T06:04:53Z

@amyeroberts Good day, this PR is ready to be merged :-)

* save NPU's RNG states when saving a checkpoint and set after all the data skip phase when resuming training. * re-trigger ci * re-trigger ci

save NPU's RNG states when saving a checkpoint and set after all the

cca3080

data skip phase when resuming training.

amyeroberts approved these changes Nov 1, 2023

View reviewed changes

muellerzr approved these changes Nov 1, 2023

View reviewed changes

statelesshz added 2 commits November 2, 2023 12:33

re-trigger ci

e012292

re-trigger ci

c89fd25

amyeroberts merged commit c52e429 into huggingface:main Nov 2, 2023
3 checks passed

statelesshz deleted the reproducible_chpt branch November 2, 2023 10:37