fix(examples:skip) Fix requirements.txt in flowertune example

[stevelaskaridis]

Issue

Description

Missing requirement in flowertune example.

Related issues/PRs

Proposal

Explanation

Add hf_transfer in requirements.txt.

Checklist

• Implement proposed change
• Write tests
• Update documentation
• Make CI checks pass
• Ping maintainers on Slack (channel #contributions)

Any other comments?

[jafermarq]

@yan-gao-GY could you take a look?

[yan-gao-GY]

Hi @stevelaskaridis, thanks for your PR! Just to confirm, I tried this example without hf_transfer package, it worked. Is this package used to speedup the downloading?

[stevelaskaridis]

Yes, seems to be that.

Crash happened on a fresh conda environment:

ray::ClientAppActor.run() (pid=163102, ip=172.17.0.2, actor_id=d3a49f36ac58159cfac8228401000000, repr=<flwr.simulation.ray_transport.ray_actor.ClientAppActor object at 0x7f51d844dbd0>)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr/simulation/ray_transport/ray_actor.py", line 57, in run
    out_message = app(message=message, context=context)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr/client/client_app.py", line 143, in __call__
    return self._call(message, context)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr/client/client_app.py", line 126, in ffn
    out_message = handle_legacy_message_from_msgtype(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr/client/message_handler/message_handler.py", line 95, in handle_legacy_message_from_msgtype
    client = client_fn(context)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr/client/client_app.py", line 67, in adaptor_fn
    return client_fn(str(cid))  # type: ignore
  File "/root/llm-flowertune/client.py", line 115, in client_fn
    else fds.load_partition(int(cid), "train")
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr_datasets/federated_dataset.py", line 169, in load_partition
    self._prepare_dataset()
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/flwr_datasets/federated_dataset.py", line 301, in _prepare_dataset
    self._dataset = datasets.load_dataset(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/load.py", line 2614, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 44, in _split_generators
    data_files = dl_manager.download_and_extract(self.config.data_files)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/download/download_manager.py", line 434, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/download/download_manager.py", line 257, in download
    downloaded_path_or_paths = map_nested(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 511, in map_nested
    mapped = [
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 512, in <listcomp>
    _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 399, in _single_map_nested
    mapped = [_single_map_nested((function, v, batched, batch_size, types, None, True, None)) for v in pbar]
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 399, in <listcomp>
    mapped = [_single_map_nested((function, v, batched, batch_size, types, None, True, None)) for v in pbar]
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 380, in _single_map_nested
    return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 380, in <listcomp>
    return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/download/download_manager.py", line 313, in _download_batched
    return [
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/download/download_manager.py", line 314, in <listcomp>
    self._download_single(url_or_filename, download_config=download_config)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/download/download_manager.py", line 323, in _download_single
    out = cached_path(url_or_filename, download_config=download_config)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 201, in cached_path
    output_path = get_from_cache(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 676, in get_from_cache
    fsspec_get(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 385, in fsspec_get
    fs.get_file(path, temp_file.name, callback=callback)
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 648, in get_file
    http_get(
  File "/opt/conda/envs/flowertune_v2/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 452, in http_get
    raise ValueError(
ValueError: Fast download using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.

After digging a bit more into it seems to be because the cloud provider image sets the HF_HUB_ENABLE_HF_TRANSFER=1 in /etc/environment. Therefore, huggingface_hub complains that this is not installed (https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/file_download.py#L443).

Feel free to not merge it as I don't think it is an actual requirement.

[yan-gao-GY]

@stevelaskaridis I see, thanks for your explanation. I think it's not bad to add this package into re.txt to avoid potential crash as above. Wdyt? @jafermarq

[jafermarq]

@stevelaskaridis thank you for the PR. I agree with @yan-gao-GY, let's add this to requirements.txt.