Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support streaming the download and extraction of zstd bundles #2538

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

henrymercer
Copy link
Contributor

We can save time installing the bundle by streaming the download and extraction phases, so that CPU time previously spent waiting on the network can now be spent extracting what we've received so far.

Some implementation notes:

  • Streaming is enabled behind a feature flag.

  • We implement streaming only for Zstandard so that in the event of a bug with streaming, we can fallback to a download first then extract method.

  • The Actions toolkit HTTP client downloads the full response body, so we can't use it to stream the extraction. Therefore, we use the lower-level Node APIs. But these don't handle redirects, so we add a dependency on follow-redirects to avoid implementing error-prone redirect handling ourselves.

  • We add some performance measurements so we can see how much faster streaming is compared to downloading first then extracting.

  • This PR also adds some logging improvements to the setup CodeQL step, removing the verbose tar extraction output, and only echoing commands for which we're streaming the output.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Confirm the readme has been updated if necessary.
  • Confirm the changelog has been updated if necessary.

@henrymercer henrymercer requested a review from a team as a code owner October 10, 2024 18:59
@henrymercer henrymercer added the Update dependencies Trigger PR workflow to update dependencies label Oct 10, 2024
@github-actions github-actions bot removed the Update dependencies Trigger PR workflow to update dependencies label Oct 10, 2024
@henrymercer
Copy link
Contributor Author

I think I'm running a newer version of npm on my machine which is reformatting the package.json files.

Copy link
Contributor

Pushed a commit to update the checked-in dependencies. Please mark the PR as ready for review to trigger PR checks.

@github-actions github-actions bot marked this pull request as draft October 10, 2024 19:22
@henrymercer henrymercer marked this pull request as ready for review October 10, 2024 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant