Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix commonpath in read-only filesystem #2073

Merged
merged 1 commit into from
Mar 1, 2024

Conversation

stevelaskaridis
Copy link
Contributor

@stevelaskaridis stevelaskaridis commented Mar 1, 2024

Description

When downloading a model, commonpath is defined as the LCA (Lowest Common Ancestor) of huggingface's cachedir and target output directory. When this ends-up being a directory that is not read-only (e.g. /), an OSError (errno: 30) is thrown.

This PR catches that error and tries to check symlink creation in the destination directory.

To reproduce

model_path = snapshot_download(repo_id=model,
                               local_dir='/tmp',
                               cache_dir=None, # TRANSFORMERS_CACHE default (i.e. ~/.cache/huggingface/hub/)
                               token=os.environ.get('HF_TOKEN', None))

Original behaviour

Tested on MacOS 14.1.2 and python 3.11.4, the output was:

OSError: [Errno 30] Read-only file system: '/tmpzeu59eox'

New behaviour

The model is persisted as intended.

@stevelaskaridis
Copy link
Contributor Author

I am not even sure why symlinks are originally tested on commonpath, but tried to stay as close as possible to the initial functionality.

Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense yes, thanks for reporting the issue and fixing it in a PR!
For the record, we might not need this fix in a near future once we revampt to_local_dir logic in #1738.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Wauplin Wauplin merged commit 05d715f into huggingface:main Mar 1, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants