Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable pulling large files in parallel #738

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

souleb
Copy link
Member

@souleb souleb commented Feb 26, 2024

This is an attempt to better meet the expectation of users that pull large files.

If implemented this will permit to pull concurrently chunks of a given artifact layer. Max concurrency is 10 at the moment.

This feature only works for registries that allow range requests.

@souleb souleb requested a review from stefanprodan as a code owner February 26, 2024 14:45
@souleb souleb marked this pull request as draft February 26, 2024 14:45
This is an attempt to better meet the expectation of users that pull
large files.

If implemented this will permit to pull concurrently chunks of a given
artifact layer.

Signed-off-by: Soule BA <bah.soule@gmail.com>
@souleb souleb mentioned this pull request Mar 4, 2024
3 tasks
Signed-off-by: Soule BA <bah.soule@gmail.com>
@souleb souleb force-pushed the pull-large-artifact-concurrently branch from 21e801e to 8fc8f0b Compare March 4, 2024 16:46
if err != nil {
return fmt.Errorf("failed to create layer file: %w", err)
}
defer file.Close()

Check warning

Code scanning / CodeQL

Writable file handle closed without error handling Warning

File handle may be writable as a result of data flow from a
call to OpenFile
and closing it may result in data loss upon failure, which is not handled explicitly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant