common: llama_load_model_from_url split support #6192
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phymbert
force-pushed
the
hp/split/load-model-from-url
branch
from
3c68a4d
to
a4a6d95
Compare
4 tasks
…tion common: llama_load_model_from_url: - fix header name case sensitive - support downloading additional split in parallel - hide password in url
phymbert
force-pushed
the
hp/split/load-model-from-url
branch
from
a4a6d95
to
ddb13ed
Compare
phymbert
commented
Mar 23, 2024
common/common.cpp
Outdated
|
|
||
| // Wait for all downloads to complete | ||
| for(auto &f : futures_download) { | ||
| if(!f.get()) { |
There was a problem hiding this comment.
Here if one download fails, other will continue in background. I think it's acceptable for the first version ?
There was a problem hiding this comment.
Yes I think, in worst case user can re-run to re-download the missing part. That reminds me about how docker download multiple layers.
In the future we can allow developer to modify this behavior, we can add a new llama_download_params to specify:
stop_immediate_on_fail==> true to stop if one part failsn_parallel==> maximum number of downloads can run in parallelforce_redownload==> skip checking etag
phymbert
commented
Mar 23, 2024
| @@ -1844,14 +1855,14 @@ struct llama_model * llama_load_model_from_url( | |||
| curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); | |||
There was a problem hiding this comment.
Here progression of each split will be flushed to stderr concurrently, it can be improved later on, the actual final result is good enough for a first version.
phymbert
commented
Mar 23, 2024
phymbert
commented
Mar 23, 2024
| } | ||
|
|
||
| curl_easy_cleanup(curl); | ||
|
|
||
| if (n_split > 1) { |
There was a problem hiding this comment.
With the current gguf-split implementation, the first file includes tensor data. Probably in future the first split should only contain metadata to trigger the tensor data download in // earlier.
phymbert
changed the title
ngxson
reviewed
Mar 23, 2024
|
@ggerganov Ubuntu Release server tests are disabled on PR, I think it was not the intention in ? Can I enable it back ? |
Yes |
Merged
ggerganov
approved these changes
Mar 23, 2024
hodlen
pushed a commit
to hodlen/llama.cpp
that referenced
this pull request
Apr 1, 2024
* llama: llama_split_prefix fix strncpy does not include string termination common: llama_load_model_from_url: - fix header name case sensitive - support downloading additional split in parallel - hide password in url * common: EOL EOF * common: remove redundant LLAMA_CURL_MAX_PATH_LENGTH definition * common: change max url max length * common: minor comment * server: support HF URL options * llama: llama_model_loader fix log * common: use a constant for max url length * common: clean up curl if file cannot be loaded in gguf * server: tests: add split tests, and HF options params * common: move llama_download_hide_password_in_url inside llama_download_file as a lambda * server: tests: enable back Release test on PR * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
hodlen
pushed a commit
to hodlen/llama.cpp
that referenced
this pull request
Apr 3, 2024
* llama: llama_split_prefix fix strncpy does not include string termination common: llama_load_model_from_url: - fix header name case sensitive - support downloading additional split in parallel - hide password in url * common: EOL EOF * common: remove redundant LLAMA_CURL_MAX_PATH_LENGTH definition * common: change max url max length * common: minor comment * server: support HF URL options * llama: llama_model_loader fix log * common: use a constant for max url length * common: clean up curl if file cannot be loaded in gguf * server: tests: add split tests, and HF options params * common: move llama_download_hide_password_in_url inside llama_download_file as a lambda * server: tests: enable back Release test on PR * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This was referenced Apr 3, 2024
tybalex
pushed a commit
to rubra-ai/tools.cpp
that referenced
this pull request
Apr 17, 2024
* llama: llama_split_prefix fix strncpy does not include string termination common: llama_load_model_from_url: - fix header name case sensitive - support downloading additional split in parallel - hide password in url * common: EOL EOF * common: remove redundant LLAMA_CURL_MAX_PATH_LENGTH definition * common: change max url max length * common: minor comment * server: support HF URL options * llama: llama_model_loader fix log * common: use a constant for max url length * common: clean up curl if file cannot be loaded in gguf * server: tests: add split tests, and HF options params * common: move llama_download_hide_password_in_url inside llama_download_file as a lambda * server: tests: enable back Release test on PR * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * spacing Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Since we can load model split in #6187, this change allows to load a model from an url composed of GGUFs split.
Changes
llama_split_prefixstrncpy includes string termination--splitand--model-url#6223Example:
logs