Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare for release v0.27 #2622

Merged
merged 5 commits into from
Oct 22, 2024
Merged

prepare for release v0.27 #2622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
436ee73
prepare for release v0.27
hanouticelina Oct 21, 2024
1378cad
Add tip
hanouticelina Oct 21, 2024
fe777a2
fix tests
hanouticelina Oct 21, 2024
c134f28
Merge branch 'main' into prepare-for-0.27
hanouticelina Oct 21, 2024
ec5123a
Merge branch 'main' into prepare-for-0.27
hanouticelina Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 1 addition & 34 deletions docs/source/en/guides/upload.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,41 +234,8 @@ Future(...)
### Upload a folder by chunks

[`upload_folder`] makes it easy to upload an entire folder to the Hub. However, for large folders (thousands of files or
hundreds of GB), it can still be challenging. If you have a folder with a lot of files, you might want to upload
it in several commits. If you experience an error or a connection issue during the upload, you would not have to resume
the process from the beginning.

To upload a folder in multiple commits, just pass `multi_commits=True` as argument. Under the hood, `huggingface_hub`
will list the files to upload/delete and split them in several commits. The "strategy" (i.e. how to split the commits)
is based on the number and size of the files to upload. A PR is open on the Hub to push all the commits. Once the PR is
ready, the commits are squashed into a single commit. If the process is interrupted before completing, you can rerun
your script to resume the upload. The created PR will be automatically detected and the upload will resume from where
it stopped. It is recommended to pass `multi_commits_verbose=True` to get a better understanding of the upload and its
progress.

The example below will upload the checkpoints folder to a dataset in multiple commits. A PR will be created on the Hub
and merged automatically once the upload is complete. If you prefer the PR to stay open and review it manually, you can
pass `create_pr=True`.
hundreds of GB), we recommend using [`upload_large_folder`], which splits the upload into multiple commits. See the [Upload a large folder](#upload-a-large-folder) section for more details.

```py
>>> upload_folder(
... folder_path="local/checkpoints",
... repo_id="username/my-dataset",
... repo_type="dataset",
... multi_commits=True,
... multi_commits_verbose=True,
... )
```

If you want a better control on the upload strategy (i.e. the commits that are created), you can have a look at the
low-level [`plan_multi_commits`] and [`create_commits_on_pr`] methods.

<Tip warning={true}>

`multi_commits` is still an experimental feature. Its API and behavior is subject to change in the future without prior
notice.

</Tip>

### Scheduled uploads

Expand Down
2 changes: 0 additions & 2 deletions docs/source/en/package_reference/hf_api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ models = hf_api.list_models()

[[autodoc]] HfApi

[[autodoc]] plan_multi_commits

## API Dataclasses

### AccessRequest
Expand Down
2 changes: 0 additions & 2 deletions docs/source/ko/package_reference/hf_api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ models = hf_api.list_models()

[[autodoc]] HfApi

[[autodoc]] plan_multi_commits

## API Dataclasses[[api-dataclasses]]

### AccessRequest[[huggingface_hub.hf_api.AccessRequest]]
Expand Down
12 changes: 1 addition & 11 deletions src/huggingface_hub/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
from typing import TYPE_CHECKING


__version__ = "0.26.0.dev0"
__version__ = "0.27.0.dev0"

# Alphabetical order of definitions is ensured in tests
# WARNING: any comment added in this dictionary definition will be lost when
Expand All @@ -70,10 +70,6 @@
"logout",
"notebook_login",
],
"_multi_commits": [
"MultiCommitException",
"plan_multi_commits",
],
"_snapshot_download": [
"snapshot_download",
],
Expand Down Expand Up @@ -167,7 +163,6 @@
"create_branch",
"create_collection",
"create_commit",
"create_commits_on_pr",
"create_discussion",
"create_inference_endpoint",
"create_pull_request",
Expand Down Expand Up @@ -601,10 +596,6 @@ def __dir__():
logout, # noqa: F401
notebook_login, # noqa: F401
)
from ._multi_commits import (
MultiCommitException, # noqa: F401
plan_multi_commits, # noqa: F401
)
from ._snapshot_download import snapshot_download # noqa: F401
from ._space_api import (
SpaceHardware, # noqa: F401
Expand Down Expand Up @@ -694,7 +685,6 @@ def __dir__():
create_branch, # noqa: F401
create_collection, # noqa: F401
create_commit, # noqa: F401
create_commits_on_pr, # noqa: F401
create_discussion, # noqa: F401
create_inference_endpoint, # noqa: F401
create_pull_request, # noqa: F401
Expand Down
Loading
Loading