Skip to content

Releases: huggingface/huggingface_hub

Patch release v0.13.3

20 Mar 12:24
@Wauplin Wauplin
v0.13.3
58d8242
Compare
Choose a tag to compare
Patch release v0.13.3

Patch to fix symlinks in the cache directory. Relative paths are used by default whenever possible. Absolute paths are used only on Windows when creating a symlink betweenh 2 paths that are not on the same volume. This hot-fix reverts the logic to what it was in huggingface_hub<=0.12 given the issues that have being reported after the 0.13.2 release (#1398, huggingface/diffusers#2729 and huggingface/transformers#22228)

Hotfix - use relative symlinks whenever possible #1399 @Wauplin

Full Changelog: v0.13.2...v0.13.3

Contributors

Wauplin
Assets 2
Loading

Patch release v0.13.2

13 Mar 17:47
@Wauplin Wauplin
v0.13.2
4bd3ddd
Compare
Choose a tag to compare
Patch release v0.13.2

Patch to fix symlinks in the cache directory. All symlinks are now absolute paths.

Full Changelog: v0.13.1...v0.13.2

Contributors

Wauplin
Assets 2
Loading

Patch release v0.13.1

09 Mar 14:24
@Wauplin Wauplin
v0.13.1
fbc4ccf
Compare
Choose a tag to compare
Patch release v0.13.1

Patch to fix upload_folder when passing path_in_repo=".". That was a breaking change compared to 0.12.1. Also added more validation around the path_in_repo attribute to improve UX.

  • Fix path_in_repo validation when committing files by @Wauplin in #1382

Full Changelog: v0.13.0...v0.13.1

Contributors

Wauplin
Assets 2
Loading

v0.13.0: download files to a specific folder, documentation, duplicate spaces, and more

08 Mar 09:10
@Wauplin Wauplin
v0.13.0
60df27b
Compare
Choose a tag to compare
v0.13.0: download files to a specific folder, documentation, duplicate spaces, and more

Download files to a specific folder

It is now possible to download files from the Hub and move them to a specific folder!

Two behaviors are possible: either create symlinks or move the files from the cache. This can be controlled with the local_dir_use_symlinks input parameter. The default -and recommended- value is "auto" which will duplicate small files to ease user experience (no symlinks when editing a file) and create symlinks for big files (save disk usage).

from huggingface_hub import snapshot_download
# or "from huggingface_hub import hf_hub_download"

# Download and cache files + duplicate small files (<5MB) to "my-folder" + add symlinks for big files
snapshot_download(repo_id, local_dir="my-folder")

# Download and cache files + add symlinks in "my-folder"
snapshot_download(repo_id, local_dir="my-folder", local_dir_use_symlinks=True)

# Duplicate files already existing in cache and/or download missing files directly to "my-folder"
snapshot_download(repo_id, local_dir="my-folder", local_dir_use_symlinks=False)

Documentation

Efforts to improve documentation have continued. The guides overview has been refactored to display which topics are covered (repository, upload, download, search, inference, community tab, cache, model cards, space management and integration).

Upload / Download files

The repository, upload and download guides have been revisited to showcase the different possibilities to manage a repository and upload/download files to/from it. The focus has been explicitly put on the HTTP endpoints rather than the git cli.

  • Refactor guides section + promote HTTP over GIT by @Wauplin in #1338

Integrate a library

A new guide has been added on how to integrate any ML framework with the Hub. It explains what is meant by that and how to do it. Here is the summary table to remember:

2023-03-07_16-32

Other

New endpoints + QOL improvements

Duplicate a Space

It's now possible to duplicate a Space programmatically!

>>> from huggingface_hub import duplicate_space

# Duplicate a Space to your account
>>> duplicate_space("multimodalart/dreambooth-training")
RepoUrl('https://huggingface.co/spaces/nateraw/dreambooth-training',...)

delete_patterns in upload_folder

New input parameter delete_patterns for the upload_folder method. It allows to delete some remote files before pushing a folder to the Hub, in a single commit. Useful when you don't exactly know which files have already been pushed. Here is an example to upload log files while deleting existing logs on the Hub:

api.upload_folder(
    folder_path="/path/to/local/folder/logs",
    repo_id="username/trained-model",
    path_in_repo="experiment/logs/",
    allow_patterns="*.txt", # Upload all local text files
    delete_patterns="*.txt", # Delete all remote text files before
)
  • Add delete_patterns option to upload_folder by @Wauplin in #1370

List repo history

Get the repo history (i.e. all the commits) for a given revision.

# Get initial commit on a repo
>>> from huggingface_hub import list_repo_commits
>>> initial_commit = list_repo_commits("gpt2")[-1]

# Initial commit is always a system commit containing the `.gitattributes` file.
>>> initial_commit
GitCommitInfo(
    commit_id='9b865efde13a30c13e0a33e536cf3e4a5a9d71d8',
    authors=['system'],
    created_at=datetime.datetime(2019, 2, 18, 10, 36, 15, tzinfo=datetime.timezone.utc),
    title='initial commit',
    message='',
    formatted_title=None,
    formatted_message=None
)
  • Add list_repo_commits to list git history of a repo by @Wauplin in #1331

Accept token in huggingface-cli login

--token and --add-to-git-credential option have been added to login directly from the CLI using an environment variable. Useful to login in a Github CI script for example.

huggingface-cli login --token $HUGGINGFACE_TOKEN --add-to-git-credential
  • Add token and git credentials to login cli command by @silvanocerza in #1372
  • token in CLI login docs by @Wauplin (direct commit on main)

Telemetry helper

Helper for external libraries to track usage of specific features of their package. Telemetry can be globally disabled by the user using HF_HUB_DISABLE_TELEMETRY.

from huggingface_hub.utils import send_telemetry

send_telemetry("gradio/local_link", library_name="gradio", library_version="3.22.1")

Breaking change

When loading a model card with an invalid model_index in the metadata, an error is explicitly raised. Previous behavior was to trigger a warning and ignore the model_index. This was problematic as it could lead to a loss of information. Fixing this is a breaking change but impact should be limited as the server is already rejecting invalid model cards. An optional ignore_metadata_errors argument (default to False) can be used to load the card with only a warning.

  • Explicit raise on invalid model_index + add ignore_metadata_errors option by @Wauplin in #1377

Bugfixes & small improvements

Model cards, datasets cards and space cards

A few improvements in repo cards: expose RepoCard as top-level, dict-like methods for RepoCardData object (#1354), updated template and improved type annotation for metadata.

  • Updating MC headings by @EziOzoani in #1367
  • Switch datasets type in ModelCard to a list of datasets by @davanstrien in #1356
  • Expose RepoCard at top level + few qol improvements by @Wauplin in #1354
  • Explicit raise on invalid model_index + add ignore_metadata_errors option by @Wauplin in #1377

Misc

Contributors

silvanocerza, albertvillanova, and 4 other contributors
Assets 2
Loading

Patch release v0.12.1

17 Feb 08:52
@Wauplin Wauplin
v0.12.1
39b8a49
Compare
Choose a tag to compare
Patch release v0.12.1

Hot-fix to remove authorization header when following redirection (using cached_download). Fix was already implemented for hf_hub_download but we forgot about this one. Has only a consequence when downloading LFS files from Spaces. Problem arose since a server-side change on how files are served. See #1345.

Full Changelog: v0.12.0...v0.12.1

Assets 2
Loading

v0.12.0: Spaces on steroids 🚀, extended HfApi, documentation, modelcards and more

20 Jan 09:16
@Wauplin Wauplin
v0.12.0
e8f0676
Compare
Choose a tag to compare
v0.12.0: Spaces on steroids 🚀, extended HfApi, documentation, modelcards and more

Spaces on steroids 🚀

Spaces support has been substantially enhanced. You can now:

  • Request hardware for your Space !
  • Configure your Space secrets
  • Get runtime information about your Space
# Assign hardware when creating the Space
api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="gradio", space_hardware="cpu-upgrade")

# Configure some secrets
api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value="hf_api_***")

# Request hardware on the fly
api.request_space_hardware(repo_id=repo_id, hardware="t4-medium")

# Get Space runtime (state, hardware, sdk,...)
api.get_space_runtime(repo_id=repo_id)

Visit the docs for more details.

And in bonus: Spaces now support Dockerfile natively !

api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="docker")

Check out the docs for more details.

New features and improvements for HfApi

List all branches and tags from a repo

It's now possible to list branches/tags from a repo, getting exact ref and target_commit.
More details in the docs.

>>> api.list_repo_refs("bigcode/the-stack", repo_type='dataset')
GitRefs(
    branches=[
        GitRefInfo(name='main', ref='refs/heads/main', target_commit='18edc1591d9ce72aa82f56c4431b3c969b210ae3'),
        GitRefInfo(name='v1.1.a1', ref='refs/heads/v1.1.a1', target_commit='f9826b862d1567f3822d3d25649b0d6d22ace714')],
    converts=[],
    tags=[
        GitRefInfo(name='v1.0', ref='refs/tags/v1.0', target_commit='c37a8cd1e382064d8aced5e05543c5f7753834da')
    ]
)

Manage your "favorite repos" list

New endpoints to like a repo, unlike it and list_liked_repos.

  • Add method to list liked repos by @Wauplin in #1254
  • Fix list_liked_repos (only public likes are returned) by @Wauplin in #1273
  • Fix like test by @Wauplin (direct commit on main)

Get the "real" repo_id when creating a repo

When using create_repo, one can provide a simple repo_name without specifying a namespace (example: "my-cool-model" instead of "Wauplin/my-cool-model"). This was annoying for as one could easily know if a namespace has been added to the provided repo_id. To facilitate this, the return value of create_repo is now an instance of RepoUrl which contains information like the endpoint, namespace, repo_id and repo_type.

Create branch from any revision

By default, new branches start from main HEAD. It's now possible to specify any branch, tag or commit id to start from.

Modelcards

Modelcards module is getting some adjustments to better integrate with the Hub. The scope of this work is larger than "just" huggingface_hub and resulted in the launch of the HF Model Card Guidebook to help operationalize model cards in the ML community.

  • Clarify what "More Resources" is by breaking it into its parts. by @meg-huggingface in #1238
  • Adding "summary" variables by @meg-huggingface in #1248
  • datasetcard_template: I think linking to a GH user does not make sense anymore now that dataset repos are fully on the Hub by @julien-c in #1257
  • Push empty model card and dataset card to the Hub by @Wauplin in #1261
  • [Dataset | Model card] When pushing to template repos, work on actual raw contents by @julien-c in #1282
  • Update REGEX_YAML_BLOCK by @julien-c in #1285
  • Make model use more visible by @osanseviero in #1286

Documentation updates

Quite some effort has been put into the documentation in the past few weeks:

Test suite for dependent libraries

huggingface_hub is getting more and more mature but you might still have some friction if you are maintainer of a library depending on huggingface_hub. To help detect breaking changes that would affect third-party libraries, we built a framework to run simple end-to-end tests in our CI. This is still preliminary work but the hope is make hfh ecosystem more and more robust over time. Check out our README for more details.

  • [RFC] Contrib test suite + tests for timm and sentence_transformers by @Wauplin in #1200

Faster download

Goal is to download files faster. First step has been to increase the chunk size by which the files are uploaded. Second step has been to add an optional Rust extension. This is not officially documented for now as we are internally assessing its benefits and limits. Installing and activating hf_transfer is purely optional.

Breaking changes

  • Repository "clone_from" feature do not create the remote repository if it doesn't exist on the Hub. Please use create_repo first before cloning it locally. The private init argument has also been removed as it was not needed anymore.
  • allow_regex and ignore_regex have been removed from snapshot_download in favor allow_patterns and ignore_patterns.
  • git-based implementation of push_to_hub_fastai has been removed in favor of the HTTP-based approach. Same for ModelHubMixin, PyTorchModelHubMixin, push_to_hub_keras and KerasModelHubMixin.
  • create_repo is now forced to use keyword-arguments. Same for metadata_eval_result.

QOL improvements

Not really some features, not really some fixes. But definitely a quality of life improvement for users 🙂

  • Progress bar when uploading LFS files by @Wauplin in #1266
  • tell user how to paste token on Windows by @fristhon in #1289
  • fixes 1290 - check for existing token during interpreter login by @lonnen in #1291
  • Store token in configurable HF_HOME (in HfFolder) by @Wauplin in #1236
  • Compute sha by chunks when uploading by @Wauplin in #1296

Internal

CI runs on Windows !

Misc

  • Remove obsolete comment by @severo in #1265
  • fix: authorization header should never forwarded by @rtrompier in #1269
  • refacto Offline repository tests by @Wauplin (direct commit on main)
  • Revert "refacto Offline repository tests" by @Wauplin (direct commit on main)
  • 1278 More robust temporary directory by @Wauplin in #1279
  • Refacto Repository tests by @Wauplin in #1277
  • skip flake8 B028 by @Wauplin (direct commit on main)

Bugfixes & small improvements

  • Fix metadata_update for verified evaluations by @lewtun in #1214
  • Accept pathlib.Path in upload_file by @Wauplin in #1247
  • Skip failing tests related to DatasetFilter by @Wauplin in #1251
  • fix tasks ids test by @Wauplin (direct commit on main)
  • Fix InferenceAPI on image task by @Wauplin in #1270
  • Update "move_repo" doc url + fix Bigcode test in CI by @Wauplin in #1292
  • Fix test_comment_discussion by @Wauplin in #1295
  • Handle hf:// urls + raise ValueError if repo...
Read more

Contributors

lonnen, Narsil, and 9 other contributors
Assets 2
Loading

Patch release v0.11.1

28 Nov 10:48
@Wauplin Wauplin
v0.11.1
cc1b547
Compare
Choose a tag to compare
Patch release v0.11.1

Hot-fix to fix permission issues when downloading with hf_hub_download or snapshot_download. For more details, see #1220, #1141 and #1215.

Full changelog: v0.11.0...v0.11.1

Assets 2
Loading

Extended HfApi, pagination, simplified login and more

14 Nov 12:00
@Wauplin Wauplin
v0.11.0
fb9f637
Compare
Choose a tag to compare
Extended HfApi, pagination, simplified login and more

New features and improvements for HfApi

HfApi is the central point to interact with the Hub API (manage repos, create commits,...). The goal is to propose more and more git-related features using HTTP endpoints to allow users to interact with the Hub without cloning locally a repo.

Create/delete tags and branches

from huggingface_hub import create_branch, create_tag, delete_branch, delete_tag

create_tag(repo_id, tag="v0.11", tag_message="Release v0.11")
delete_tag(repo_id, tag="something") # If you created a tag by mistake

create_branch(repo_id, branch="experiment-154")
delete_branch(repo_id, branch="experiment-1") # Clean some old branches

Upload lots of files in a single commit

Making a very large commit was previously tedious. Files are now processed by chunks which makes it possible to upload 25k files in a single commit (and 1Gb payload limitation if uploading only non-LFS files). This should make it easier to upload large datasets.

  • Create commit by streaming a ndjson payload (allow lots of file in single commit) by @Wauplin in #1117

Delete an entire folder

from huggingface_hub import CommitOperationDelete, create_commit, delete_folder

# Delete a single folder
delete_folder(repo_id=repo_id, path_in_repo="logs/")

# Alternatively, use the low-level `create_commit`
create_commit(
    repo_id,
    operations=[
        CommitOperationDelete(path_in_repo="old_config.json") # Delete a file
        CommitOperationDelete(path_in_repo="logs/") # Delete a folder
    ],
    commit_message=...,
)

Support pagination when listing repos

In the future, listing models, datasets and spaces will be paginated on the Hub by default. To avoid breaking changes, huggingface_hub follows already pagination. Output type is currently a list (deprecated), will become a generator in v0.14.

  • Add support for pagination in list_models list_datasets and list_spaces by @Wauplin #1176
  • Deprecate output in list_models by @Wauplin in #1143

Misc

  • Allow create PR against non-main branch by @Wauplin in #1168
  • 1162 Reorder operations correctly in commit endpoint by @Wauplin in #1175

Login, tokens and authentication

Authentication has been revisited to make it as easy as possible for the users.

Unified login and logout methods

from huggingface_hub import login, logout

# `login` detects automatically if you are running in a notebook or a script
# Launch widgets or TUI accordingly
login()

# Now possible to login with a hardcoded token (non-blocking)
login(token="hf_***")

# If you want to bypass the auto-detection of `login`
notebook_login()  # still available
interpreter_login()  # to login from a script

# Logout programmatically
logout()
# Still possible to login from CLI
huggingface-cli login

Set token only for a HfApi session

from huggingface_hub import HfApi

# Token will be sent in every request but not stored on machine
api = HfApi(token="hf_***")

Stop using use_auth_token in favor of token, everywhere

token parameter can now be passed to every method in huggingface_hub. use_auth_token is still accepted where it previously existed but the mid-term goal (~6 months) is to deprecate and remove it.

  • Replace use_auth_token arg by token everywhere by @Wauplin in #1122

Respect git credential helper from the user

Previously, token was stored in the git credential store. Can now be in any helper configured by the user -keychain, cache,...-.

  • Refactor git credential handling in login workflow by @Wauplin in #1138

Better error handling

Helper to dump machine information

# Dump all relevant information. To be used when reporting an issue.
➜ huggingface-cli env

Copy-and-paste the text below in your GitHub issue.

- huggingface_hub version: 0.11.0.dev0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
...

Misc

Modelcards

Few improvements/fixes in the modelcard module:

Cache assets

New feature to provide a path in the cache where any downstream library can store assets (processed data, files from the web, extracted data, rendered images,...)

  • [RFC] Proposal for a way to cache files in downstream libraries by @Wauplin in #1088

Documentation updates

Breaking changes

  • Cannot provide an organization to create_repo
  • identical_ok removed in upload_file
  • Breaking changes in arguments for validate_preupload_info, prepare_commit_payload, _upload_lfs_object (internal helpers for the commit API)
  • huggingface_hub.snapshot_download is not exposed as a public module anymore

Deprecations

  • Remove deprecated code from v0.9, v0.10 and v0.11 by @Wauplin in #1092
  • Rename languages to langage + remove duplicate code in tests by @Wauplin in #1169
  • Deprecate output in list_models by @Wauplin in #1143
  • Set back feature to create a repo when using clone_from by @Wauplin in #1187

Internal

Bugfixes & small improvements

Contributors

singingwolfboy, Narsil, and 9 other contributors
Assets 2
Loading

Patch release v0.10.1

11 Oct 07:48
@Wauplin Wauplin
v0.10.1
c87d402
Compare
Choose a tag to compare
Patch release v0.10.1

Hot-fix to force utf-8 encoding in modelcards. See #1102 and skops-dev/skops#162 (comment) for context.

Full Changelog: v0.10.0...v0.10.1

Assets 2
Loading

v0.10.0: Modelcards, cache management and more

28 Sep 07:26
@Wauplin Wauplin
v0.10.0
583d9f7
Compare
Choose a tag to compare
v0.10.0: Modelcards, cache management and more

Modelcards

Contribution from @nateraw to integrate the work done on Modelcards and DatasetCards (from nateraw/modelcards) directly in huggingface_hub.

>>> from huggingface_hub import ModelCard

>>> card = ModelCard.load('nateraw/vit-base-beans')
>>> card.data.to_dict()
{'language': 'en', 'license': 'apache-2.0', 'tags': ['generated_from_trainer', 'image-classification'],...}

Related commits

Related documentation

Cache management (huggingface-cli scan-cache and huggingface-cli delete-cache)

New commands in huggingface-cli to scan and delete parts of the cache. Goal is to manage the cache-system the same way for any dependent library that uses huggingface_hub. Only the new cache-system format is supported.

➜ huggingface-cli scan-cache
REPO ID                     REPO TYPE SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED REFS                LOCAL PATH
--------------------------- --------- ------------ -------- ------------- ------------- ------------------- -------------------------------------------------------------------------
glue                        dataset         116.3K       15 4 days ago    4 days ago    2.4.0, main, 1.17.0 /home/wauplin/.cache/huggingface/hub/datasets--glue
google/fleurs               dataset          64.9M        6 1 week ago    1 week ago    refs/pr/1, main     /home/wauplin/.cache/
(...)

Done in 0.0s. Scanned 6 repo(s) for a total of 3.4G.
Got 1 warning(s) while scanning. Use -vvv to print details.

Related commits

Related documentation

Better error handling (and http-related stuff)

HTTP calls to the Hub have been harmonized to behave the same across the library.

Major differences are:

  • Unified way to handle HTTP errors using hf_raise_for_status (more informative error message)
  • Auth token is always sent by default when a user is logged in (see documentation).
  • package versions are sent as user-agent header for telemetry (python, huggingface_hub, tensorflow, torch,...). It was already the case for hf_hub_download.

Related commits

  • Always send the cached token when user is logged in by @Wauplin in #1064
  • Add user agent to all requests with huggingface_hub version (and other) by @Wauplin in #1075
  • [Repository] Add better error message by @patrickvonplaten in #993
  • Clearer HTTP error messages in huggingface_hub by @Wauplin in #1019
  • Handle backoff on HTTP 503 error when pushing repeatedly by @Wauplin in #1038

Breaking changes

  1. For consistency, the return type of create_commit has been modified. This is a breaking change, but we hope the return type of this method was never used (quite recent and niche output type).
  • Return more information in create_commit output by @Wauplin in #1066
  1. Since repo_id is now validated using @validate_hf_hub_args (see below), a breaking change can be caused if repo_id was previously miused. A HFValidationError is now raised if repo_id is not valid.

Miscellaneous improvements

Add support for autocomplete

http-based push_to_hub_fastai

  • Add changes for push_to_hub_fastai to use the new http-based approach. by @nandwalritik in #1040

Check if a file is cached

  • try_to_load_from_cache returns cached non-existence by @sgugger in #1039

Get file metadata (commit hash, etag, location) without downloading

  • Add get_hf_file_metadata to fetch metadata from the Hub by @Wauplin in #1058

Validate arguments using @validate_hf_hub_args

  • Add validator for repo id + decorator to validate arguments in huggingface_hub by @Wauplin in #1029
  • Remove repo_id validation in hf_hub_url and hf_hub_download by @Wauplin in #1031

⚠️ This is a breaking change if repo_id was previously misused ⚠️

Related documentation:

Documentation updates

Deprecations

  • ENH Deprecate clone_from behavior by @merveenoyan in #952
  • 🗑 Deprecate token in read-only methods of HfApi in favor of use_auth_token by @SBrandeis in #928
  • Remove legacy helper 'install_lfs_in_userspace' by @Wauplin in #1059
  • 1055 deprecate private and repo type in repository class by @Wauplin in #1057

Bugfixes & small improvements

  • Consider empty subfolder as None in hf_hub_url and hf_hub_download by @Wauplin in #1021
  • enable http request retry under proxy by @MrZhengXin in #1022
  • Add securityStatus to ModelInfo object with default value None. by @Wauplin in #1026
  • 👽️ Add size parameter for lfsFiles when committing on the hub by @coyotte508 in #1048
  • Use /models/ path for api call to update settings by @Wauplin in #1049
  • Globally set git credential.helper to store in google colab by @Wauplin in #1053
  • FIX notebook login by @Wauplin in #1073

Windows-specific bug fixes

  • Fix default cache on windows by @thomwolf in #1069
  • Degraded but fully working cache-system when symlinks are not supported by @Wauplin in #1067
  • Check symlinks support per directory instead of globally by @Wauplin in #1077

Contributors

coyotte508, thomwolf, and 10 other contributors
Assets 2
Loading