Degraded but fully working cache-system when symlinks are not supported

[Wauplin]

Related to #1062.

The proposed approach is to use the new cache system on every platform but have a "degraded" version when symlinks are not supported. The degraded version do not use symlinks but rather copypaste/move the blob files instead. Drawback is that some files might get duplicated in the cache if several revisions of the same repo are downloaded. This drawback can be mitigated by deleting older revisions (see huggingface-cli delete-cache command #1025). The advantage of this solution is that it should work seamlessly from a user perspective but also for developers (e.g. when downloading a repo, the repo structure will still be the same on disk)

A warning will be displayed to encourage the user to enable developer mode/run as admin for a better cache system. Warning message can be disabled via a environment variable.

Todo:

• detect platform that doesn't support symlinks
• move/copypaste file when symlinks are not supported
• test downloading a file when symlinks not supported
• test scan cache + delete cache when symlinks are not supported (or even half-supported)
• documentation

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[LysandreJik]

Sounds approach, only left a comment

[codecov]

Codecov Report

Base: 84.16% // Head: 84.36% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (83da9e8) compared to base (3a59bf1).
Patch coverage: 96.15% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1067      +/-   ##
==========================================
+ Coverage   84.16%   84.36%   +0.19%     
==========================================
  Files          40       40              
  Lines        4087     4100      +13     
==========================================
+ Hits         3440     3459      +19     
+ Misses        647      641       -6     

Impacted Files	Coverage Δ
src/huggingface_hub/utils/_cache_manager.py	96.55% <ø> (+0.77%)	⬆️
src/huggingface_hub/file_download.py	88.49% <96.15%> (+1.62%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[LysandreJik]

Ok this sounds good to me. I don't have a windows setup handy right now but I can test on one later tonight.

[Wauplin]

@LysandreJik that would be perfect thank you 🤗

[LysandreJik]

I just checked on Windows by first disabling developer mode. Using v0.9.1 or main, I get the following error:

>>> from huggingface_hub import snapshot_download
>>> snapshot_download('distilbert-base-uncased')

Downloading: 100%|█████████████████████████████████████████████████████████████████████| 391/391 [00:00<00:00, 392kB/s]
Fetching 11 files:   0%|                                                                        | 0/11 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\file_download.py", line 879, in _create_relative_symlink
    os.symlink(relative_src, dst)
OSError: [WinError 1314] A required privilege is not held by the client: '..\\..\\blobs\\ae8c63daedbd4206d7d40126955d4e6ab1c80f8f' -> 'C:\\Users\\Lysandre\\.cache\\huggingface\\hub\\models--distilbert-base-uncased\\snapshots\\043235d6088ecd3dd5fb5ca3592b6913fd516027\\.gitattributes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\utils\_deprecation.py", line 93, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\utils\_validators.py", line 90, in _inner_fn
    return fn(*args, **kwargs)
  File "C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\_snapshot_download.py", line 197, in snapshot_download
    _ = hf_hub_download(
  File "C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\file_download.py", line 1327, in hf_hub_download
    _create_relative_symlink(blob_path, pointer_path)
  File "C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\file_download.py", line 883, in _create_relative_symlink
    raise OSError(
OSError: Windows requires Developer Mode to be activated, or to run Python as an administrator, in order to create symlinks.
In order to activate Developer

Checking out this current branch and running the code:

>>> from huggingface_hub import snapshot_download
>>> snapshot_download('distilbert-base-cased')

Fetching 8 files:   0%|                                                                          | 0/8 [00:00<?, ?it/s]C:\Users\Lysandre\PycharmProjects\huggingface_hub\src\huggingface_hub\file_download.py:214: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine doesn't support them. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
Downloading: 100%|████████████████████████████████████████████████████████████████| 8.79k/8.79k [00:00<00:00, 8.86MB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████| 411/411 [00:00<00:00, 416kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████| 263M/263M [00:32<00:00, 8.18MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████| 354M/354M [00:41<00:00, 8.45MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████| 436k/436k [00:00<00:00, 1.35MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████| 29.0/29.0 [00:00<00:00, 29.0kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████| 213k/213k [00:00<00:00, 2.89MB/s]
Fetching 8 files: 100%|██████████████████████████████████████████████████████████████████| 8/8 [01:16<00:00,  9.56s/it]
'C:\\Users\\Lysandre\\.cache\\huggingface\\hub\\models--distilbert-base-cased\\snapshots\\8d708decd7afb7bec0af233e5338fe1fca3db705'

It seems it works! Any other situation you'd like me to test?

[Wauplin]

Thanks a lot @LysandreJik, that's reassuring ! Just to be sure, could you also try huggingface-cli scan-cache and huggingface-cli delete-cache ?
Otherwise I think it's safe to assume we can merge :)

[LysandreJik]

I confirm both scan-cache and delete-cache work on Windows as well!

[Wauplin]

Thanks @LysandreJik !
(FYI, I started a windows machine to be able to run tests myself next time we need it ;) )

[LysandreJik]

To be honest this should all be taken care of by running our CI on Windows.
I think it's 95% compatible, but I bet there are some issues with paths.

I had started here https://github.com/huggingface/huggingface_hub/tree/ci_windows-tests but never finished (it's already been 14 months 😅 ...)

[Wauplin]

@LysandreJik yep true. Let's create an issue for that #1074