Add support for resumable downloads; clean up ingest directory; add --concurrency flag
#408
Commits on Jul 16, 2024
-
Implement our own PullModel function to control how storage is used
Add basic implementation for PullModel, which pulls a modelkit from a remote reference. This is basically equivalent to the Copy() utility function in the oras library, which uses Fetch() and Push(); however, this base allows us control over how files are saved and enables implementing better cleanup and resumable downloads.
-
Implement resumable downloads when pulling modelkits
If a remote response with Accept-Ranges: "bytes", treat downloads as resumable. In this case: * Temporary ingest file is saved with the digest name in ingest directory * If a file already exists, seek to the end of it and seek to that point in the download * If a download is cancelled or encounters an error, the ingest file is not removed To avoid excessive slowdown, instead of seeking to the end of the file, we read it into a digester to resume calculating the digest of the downloaded data at the same time.
-
Clean up ingest directory on successful pull
If a pull is cancelled via signal (instead of an error), the ingest file will not be cleaned up. To avoid accrual of partially-downloaded ingest files (generated by the non-resumable download case), clear the ingest directory on every successful pull. This may remove a partially-completed resumable download, but this should only occur if the user downloads some _other_ modelkit before attempting to resume the download.
-
Reimplement progress bars for pull
Since we're no longer using a basic copy operation, we need to redo progress bars for pull from scratch.
-
-
-
Add --concurrency flag for push and pull
Add flag --concurrency to allow configuring maximum number of simultaneous uploads/downloads. Default value is 5.
-
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.