Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't deploy Rust docs to GH pages #3731

Closed
3 tasks
LesnyRumcajs opened this issue Nov 22, 2023 · 5 comments
Closed
3 tasks

Don't deploy Rust docs to GH pages #3731

LesnyRumcajs opened this issue Nov 22, 2023 · 5 comments
Labels
Priority: 3 - Medium Nice-to-have, does not impede core functionality Type: Enhancement

Comments

@LesnyRumcajs
Copy link
Member

Issue summary

Self-hosting Rust docs makes regular Forest clones/forks quite heavy; 419 MB for latest main b7ac403.

The largest culprit is the Rust documentation.

Use this command

git rev-list --objects --all |   git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |   sed -n 's/^blob //p' |   sort --numeric-sort --key=2 |   cut -c 1-12,41- |   $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest | col

Sample output:

...
2e6ab4cd5af  4.8MiB rustdoc/implementors/core/panic/unwind_safe/trait.RefUnwindSafe.js
fcc469f254c8  4.8MiB rustdoc/implementors/core/panic/unwind_safe/trait.RefUnwindSafe.js
3a3a46cdb2c0  4.8MiB rustdoc/implementors/core/panic/unwind_safe/trait.RefUnwindSafe.js
46c9f562f2cc  4.8MiB rustdoc/implementors/core/panic/unwind_safe/trait.RefUnwindSafe.js
003aba4fb25f  4.8MiB rustdoc/implementors/core/panic/unwind_safe/trait.RefUnwindSafe.js
5baeacce618e  4.8MiB rustdoc/implementors/core/panic/unwind_safe/trait.RefUnwindSafe.js
...
df90f0b30cc4   16MiB rustdoc/search-index.js
e8500f3b8af1   16MiB rustdoc/search-index.js
05e123abbb51   16MiB rustdoc/search-index.js
6cebbd16bff8   16MiB rustdoc/search-index.js
a74c4b64e5c4   16MiB rustdoc/search-index.js
c830e0d1c911   16MiB rustdoc/search-index.js
4cb635118402   16MiB rustdoc/search-index.js
ba3b8b29d3d9   16MiB rustdoc/search-index.js
31f710c25df8   17MiB rustdoc/search-index.js

This is too large and must be dealt with.

  • Remove deploying Rust docs to GH pages
  • Cleanup the gh-pages branch - most likely it will need to be squashed (or re-created) so that the objects are no longer present in git history,
  • Verify that the clone/fork size is significantly smaller than it was before.

Other information and links

@aatifsyed
Copy link
Contributor

Other big rust projects must face this issue - I'll have a look at how e.g servo addresses this

See also rust-lang/rust#31387

@aatifsyed
Copy link
Contributor

Known issue: https://www.reddit.com/r/rust/comments/wy3j50/psa_if_youre_using_ghpages_to_host_your/

As a first-step measure, we changed the CI script to overwrite the gh-pages branch at every run, rather than just appending a new commit. We use this gh-pages action, so it was just a matter of adding a force_orphan: true parameter.

Turns out that this improved the situation a lot: when it no longer needs to keep history, git manages to compress that 220MB of documentation very well, and now the whole Smithay git repo is only ~15 MB!

@aatifsyed
Copy link
Contributor

We currently use JamesIves/github-pages-deploy-action, which looks like it commits to a gh-pages branch?

- name: Deploy GH Pages
uses: JamesIves/github-pages-deploy-action@v4.4.3
if: ${{ github.event_name == 'push' }}
with:
branch: gh-pages
folder: ./documentation/book/html/
dry-run: false

Maybe switching to actions/deploy-pages would be better, or maybe the fork mentioned in the reddit thread

@ansermino
Copy link
Member

I would suggest exploring using Cloudflare pages for this, as well as pruning the git history. Cloning the full repo sucks 😁

@lemmih lemmih added Priority: 3 - Medium Nice-to-have, does not impede core functionality Type: Enhancement labels Aug 2, 2024
@LesnyRumcajs
Copy link
Member Author

Done via #4792

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: 3 - Medium Nice-to-have, does not impede core functionality Type: Enhancement
Projects
Status: Done
Development

No branches or pull requests

4 participants