From a27cce621a3fdd1b68bb3bcfb9bad7ee2155da4c Mon Sep 17 00:00:00 2001 From: James Braza Date: Fri, 12 Jan 2024 03:06:16 -0800 Subject: [PATCH] Added `hf_transfer` extra into `setup.py` and `docs/` (#1970) * Added hf_transfer extra * Updated docs to mention hf_transfer --- docs/source/en/guides/download.md | 20 +++++++++++-------- docs/source/en/guides/upload.md | 15 +++++++++----- .../environment_variables.md | 13 +++++++++++- setup.py | 4 +++- 4 files changed, 37 insertions(+), 15 deletions(-) diff --git a/docs/source/en/guides/download.md b/docs/source/en/guides/download.md index 459cdd7130..a8aa1ddab0 100644 --- a/docs/source/en/guides/download.md +++ b/docs/source/en/guides/download.md @@ -187,16 +187,20 @@ For more details about the CLI download command, please refer to the [CLI guide] ## Faster downloads -If you are running on a machine with high bandwidth, you can increase your download speed with [`hf_transfer`](https://github.com/huggingface/hf_transfer), a Rust-based library developed to speed up file transfers with the Hub. To enable it, install the package (`pip install hf_transfer`) and set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. +If you are running on a machine with high bandwidth, +you can increase your download speed with [`hf_transfer`](https://github.com/huggingface/hf_transfer), +a Rust-based library developed to speed up file transfers with the Hub. +To enable it: - - -Progress bars are supported in `hf_transfer` starting from version `0.1.4`. Consider upgrading (`pip install -U hf-transfer`) if you plan to enable faster downloads. - - +1. Specify the `hf_transfer` extra when installing `huggingface_hub` + (e.g. `pip install huggingface_hub[hf_transfer]`). +2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. -`hf_transfer` is a power user tool! It is tested and production-ready, but it lacks user-friendly features like advanced error handling or proxies. For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer). +`hf_transfer` is a power user tool! +It is tested and production-ready, +but it lacks user-friendly features like advanced error handling or proxies. +For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer). - \ No newline at end of file + diff --git a/docs/source/en/guides/upload.md b/docs/source/en/guides/upload.md index ab6483b957..7bcdf964ee 100644 --- a/docs/source/en/guides/upload.md +++ b/docs/source/en/guides/upload.md @@ -433,13 +433,18 @@ servers. For example, if you plan to upload a large number of files, it's best t already uploaded before uploading the next batch. You are ensured that an LFS file that is already committed will never be re-uploaded twice but checking it client-side can still save some time. - **Use `hf_transfer`**: this is a Rust-based [library](https://github.com/huggingface/hf_transfer) meant to speed up -uploads on machines with very high bandwidth. To use it, you must install it (`pip install hf_transfer`) and enable it -by setting `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. You can then use `huggingface_hub` normally. -Disclaimer: this is a power user tool. It is tested and production-ready but lacks user-friendly features like advanced error handling or proxies. For more details, please refer to this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer). + uploads on machines with very high bandwidth. To use `hf_transfer`: - + 1. Specify the `hf_transfer` extra when installing `huggingface_hub` + (e.g. `pip install huggingface_hub[hf_transfer]`). + 2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. + + -Progress bars are supported in `hf_transfer` starting from version `0.1.4`. Consider upgrading (`pip install -U hf-transfer`) if you plan to enable faster uploads. +`hf_transfer` is a power user tool! +It is tested and production-ready, +but it lacks user-friendly features like advanced error handling or proxies. +For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer). diff --git a/docs/source/en/package_reference/environment_variables.md b/docs/source/en/package_reference/environment_variables.md index c89f65d081..a44a52ca59 100644 --- a/docs/source/en/package_reference/environment_variables.md +++ b/docs/source/en/package_reference/environment_variables.md @@ -137,7 +137,18 @@ You can set `HF_HUB_DISABLE_TELEMETRY=1` as environment variable to globally dis Set to `True` for faster uploads and downloads from the Hub using `hf_transfer`. -By default, `huggingface_hub` uses the Python-based `requests.get` and `requests.post` functions. Although these are reliable and versatile, they may not be the most efficient choice for machines with high bandwidth. [`hf_transfer`](https://github.com/huggingface/hf_transfer) is a Rust-based package developed to maximize the bandwidth used by dividing large files into smaller parts and transferring them simultaneously using multiple threads. This approach can potentially double the transfer speed. To use `hf_transfer`, you need to install it separately [from PyPI](https://pypi.org/project/hf-transfer/) and set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. +By default, `huggingface_hub` uses the Python-based `requests.get` and `requests.post` functions. +Although these are reliable and versatile, +they may not be the most efficient choice for machines with high bandwidth. +[`hf_transfer`](https://github.com/huggingface/hf_transfer) is a Rust-based package developed to +maximize the bandwidth used by dividing large files into smaller parts +and transferring them simultaneously using multiple threads. +This approach can potentially double the transfer speed. +To use `hf_transfer`: + +1. Specify the `hf_transfer` extra when installing `huggingface_hub` + (e.g. `pip install huggingface_hub[hf_transfer]`). +2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. Please note that using `hf_transfer` comes with certain limitations. Since it is not purely Python-based, debugging errors may be challenging. Additionally, `hf_transfer` lacks several user-friendly features such as resumable downloads and proxies. These omissions are intentional to maintain the simplicity and speed of the Rust logic. Consequently, `hf_transfer` is not enabled by default in `huggingface_hub`. diff --git a/setup.py b/setup.py index 3269668911..1253e5ae07 100644 --- a/setup.py +++ b/setup.py @@ -41,7 +41,9 @@ def get_version() -> str: extras["torch"] = [ "torch", ] - +extras["hf_transfer"] = [ + "hf_transfer>=0.1.4", # Pin for progress bars +] extras["fastai"] = [ "toml", "fastai>=2.4",