Added hf_transfer extra into setup.py and docs/ (#1970)

* Added hf_transfer extra

* Updated docs to mention hf_transfer

docs/source/en/guides/download.md

@@ -187,16 +187,20 @@ For more details about the CLI download command, please refer to the [CLI guide]
 
 ## Faster downloads
 
-If you are running on a machine with high bandwidth, you can increase your download speed with [`hf_transfer`](https://github.com/huggingface/hf_transfer), a Rust-based library developed to speed up file transfers with the Hub. To enable it, install the package (`pip install hf_transfer`) and set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable.
+If you are running on a machine with high bandwidth,
+you can increase your download speed with [`hf_transfer`](https://github.com/huggingface/hf_transfer),
+a Rust-based library developed to speed up file transfers with the Hub.
+To enable it:
 
-<Tip>
-
-Progress bars are supported in `hf_transfer` starting from version `0.1.4`. Consider upgrading (`pip install -U hf-transfer`) if you plan to enable faster downloads.
-
-</Tip>
+1. Specify the `hf_transfer` extra when installing `huggingface_hub`
+ (e.g. `pip install huggingface_hub[hf_transfer]`).
+2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable.
 
 <Tip warning={true}>
 
-`hf_transfer` is a power user tool! It is tested and production-ready, but it lacks user-friendly features like advanced error handling or proxies. For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer).
+`hf_transfer` is a power user tool!
+It is tested and production-ready,
+but it lacks user-friendly features like advanced error handling or proxies.
+For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer).
 
-</Tip>
+</Tip>

docs/source/en/guides/upload.md

@@ -433,13 +433,18 @@ servers. For example, if you plan to upload a large number of files, it's best t
 already uploaded before uploading the next batch. You are ensured that an LFS file that is already committed will never
 be re-uploaded twice but checking it client-side can still save some time.
 - **Use `hf_transfer`**: this is a Rust-based [library](https://github.com/huggingface/hf_transfer) meant to speed up
-uploads on machines with very high bandwidth. To use it, you must install it (`pip install hf_transfer`) and enable it
-by setting `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable. You can then use `huggingface_hub` normally.
-Disclaimer: this is a power user tool. It is tested and production-ready but lacks user-friendly features like advanced error handling or proxies. For more details, please refer to this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer).
+ uploads on machines with very high bandwidth. To use `hf_transfer`:
 
-<Tip>
+ 1. Specify the `hf_transfer` extra when installing `huggingface_hub`
+ (e.g. `pip install huggingface_hub[hf_transfer]`).
+ 2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable.
+
+<Tip warning={true}>
 
-Progress bars are supported in `hf_transfer` starting from version `0.1.4`. Consider upgrading (`pip install -U hf-transfer`) if you plan to enable faster uploads.
+`hf_transfer` is a power user tool!
+It is tested and production-ready,
+but it lacks user-friendly features like advanced error handling or proxies.
+For more details, please take a look at this [section](https://huggingface.co/docs/huggingface_hub/hf_transfer).
 
 </Tip>
 

docs/source/en/package_reference/environment_variables.md

@@ -137,7 +137,18 @@ You can set `HF_HUB_DISABLE_TELEMETRY=1` as environment variable to globally dis
 
 Set to `True` for faster uploads and downloads from the Hub using `hf_transfer`.
 
-By default, `huggingface_hub` uses the Python-based `requests.get` and `requests.post` functions. Although these are reliable and versatile, they may not be the most efficient choice for machines with high bandwidth. [`hf_transfer`](https://github.com/huggingface/hf_transfer) is a Rust-based package developed to maximize the bandwidth used by dividing large files into smaller parts and transferring them simultaneously using multiple threads. This approach can potentially double the transfer speed. To use `hf_transfer`, you need to install it separately [from PyPI](https://pypi.org/project/hf-transfer/) and set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable.
+By default, `huggingface_hub` uses the Python-based `requests.get` and `requests.post` functions.
+Although these are reliable and versatile,
+they may not be the most efficient choice for machines with high bandwidth.
+[`hf_transfer`](https://github.com/huggingface/hf_transfer) is a Rust-based package developed to
+maximize the bandwidth used by dividing large files into smaller parts
+and transferring them simultaneously using multiple threads.
+This approach can potentially double the transfer speed.
+To use `hf_transfer`:
+
+1. Specify the `hf_transfer` extra when installing `huggingface_hub`
+ (e.g. `pip install huggingface_hub[hf_transfer]`).
+2. Set `HF_HUB_ENABLE_HF_TRANSFER=1` as an environment variable.
 
 Please note that using `hf_transfer` comes with certain limitations. Since it is not purely Python-based, debugging errors may be challenging. Additionally, `hf_transfer` lacks several user-friendly features such as resumable downloads and proxies. These omissions are intentional to maintain the simplicity and speed of the Rust logic. Consequently, `hf_transfer` is not enabled by default in `huggingface_hub`.
 

setup.py

@@ -41,7 +41,9 @@ def get_version() -> str:
 extras["torch"] = [
  "torch",
 ]
-
+extras["hf_transfer"] = [
+ "hf_transfer>=0.1.4", # Pin for progress bars
+]
 extras["fastai"] = [
  "toml",
  "fastai>=2.4",