Tracking: migrate doc build artifacts from github to hf hub

[mishig25]

Tracking: migrate doc build artifacts from github to hf hub

Why: because hf hub can handle large files & huggingface_hub python client library makes it easy to work with. (With github, we had to almost write python client from scratch using github graphql)

Dev Steps:

• Create hf hub org & repos https://huggingface.co/hf-doc-build
• Implement doc-builder changes Use huggingface hub for doc builds #315
  • test on hf-docs [wip: test doc-build] hub-docs#449
  • test on transformers [wip test doc build] transformers#19867
  • test on transformers main [wip doc buidler test] transformers#20092
  • test deletion, by closing a pr build
• Implement hub backend changes https://github.com/huggingface/moon-landing/pull/4249
• Handle credentials/tokens
  • update secrets.HUGGINGFACE_PUSH token on all repos (for example, here)
  • doc-build-pr.yml will be using secrets.HUGGINGFACE_PUSH as well. Therefore, all the repos should update its doc-build-pr.yml tp supply the token
• Migrate/copy all existing docs from github to hf hub
• Revert dev log changes in Use huggingface hub for doc builds #315

Migration Steps:

• Revert dev changes in Use huggingface hub for doc builds #315 : use token & revert branch to main
• in hub backend, rm -rf doc-build doc-build-dev
• run sync-github-repos.js

Resources:

internal slack discussion

[coyotte508]

If I understand it correctly, for now the repo structure stays the same (no LFS files, no zip), this issue is just moving the doc build repos from github to HF hub

[mishig25]

@coyotte508 at the moment, there is no zip. Would I need LFS ?, and if so, is there anything I need to do besides this step ?

[coyotte508]

LFS will make the git repos a lot smaller in size so it should be beneficial in any case.

I didn't see the gitattributes :) What's left I guess is on the hub side, making sure to git lfs install after cloning the repos.

[mishig25]

making sure to git lfs install after cloning the repos.

did so on the hub backend side 👍

[mishig25]

@coyotte508 I've actually switched to using zipped file.
When I tried on transformers docs builds (folder with files combined 95MB), hub_client was failing with:

File "/usr/local/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line [21](https://github.com/huggingface/transformers/actions/runs/3319951930/jobs/5486173258#step:11:22)3, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/datasets/hf-doc-build/doc-build-dev.git/info/lfs/objects/batch

cc: @Wauplin

When I zipped it into a zip file (14 MB), everything worked. You can find on the hub here

[mishig25]

all done