[Question] What is the best workaround for using uv when dependency is not resolvable? (e.g. PyTorch v2.4.0 + segment-anything-2 (sam-2) v1.0) #6907
Comments
|
I think https://docs.astral.sh/uv/reference/settings/#override-dependencies is the feature intended to help you there. Add a section of override dependencies to the pyproject.toml (root of the workspace if it's a workspace), to set a specific tqdm version. |
|
Thank you, @bluss! I've tried |
|
https://github.com/facebookresearch/segment-anything-2.git has torch as a build requirement, I guess I would try to use no build isolation then https://docs.astral.sh/uv/concepts/projects/#build-isolation Maybe that could help, and maybe the invisible tqdm dependency is in a build-dep? If you need more information, enable more log output when synching or when adding segment anything, such as using -v -vv or -vvv on the relevant command to get more logs. |
|
Thank you so much! I will try it later 🙌 |
|
Ok the tqdm error is also from the fact that tqdm-4.64.1-py2.py3-none-any.whl is the only version of tqdm available from that index. So that's the reason no other tqdm dependency is visible, there is none. It's a conflict between versions offered by the index and versions required by the project. override can work if it overrides to that exact version that is available, or look at uv sync --index-strategy -- but I don't know those parts very well, someone else might know. |
|
The two step sync described in docs can make this project installable though - but note as above that there are maybe other better solutions for tqdm. [project]
name = "uvtorchexample"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
]
[project.optional-dependencies]
build = [
"torch==2.4.0+cu118",
"torchvision==0.19.0+cu118",
"setuptools",
"wheel",
]
sam = [
"sam-2",
]
[tool.uv]
extra-index-url = ["https://download.pytorch.org/whl/cu118"]
override-dependencies = ["tqdm>=4.64.1"]
no-build-isolation-package = ["sam-2"]
[tool.uv.sources]
sam-2 = { path = "segment-anything-2", editable = true } |
|
@bluss I'm sorry for not getting back to you sooner. Am I correct in understanding that I should prepare the above That did not work. For some reason, at The
However, Is something wrong with my two-step sync method? If so, I would like to know the possible cause. Thanks for the great support! Here are the `uv sync --extra sam-2 -vvv` Details
|
|
Oh, it was my mistake... I should run |
One thing I would like to see |
|
Finally, here is the log of successfully getting the segment-anything-2 demo notebooks to work with uv. Thank you very much for your valuable time! @bluss ❯ uv --version
uv 0.4.3
❯ uv init test_sam2_with_uv_no_build_deps -p 3.12
Initialized project `test-sam2-with-uv-no-build-deps` at `/home/hideakitai/develop/ml/sandbox/test_sam2_with_uv_no_build_deps`
❯ cd test_sam2_with_uv_no_build_deps/
❯ uv python pin 3.12.5
Pinned `.python-version` to `3.12.5`
❯ git clone https://github.com/facebookresearch/segment-anything-2.git
Cloning into 'segment-anything-2'...
remote: Enumerating objects: 510, done.
remote: Total 510 (delta 0), reused 0 (delta 0), pack-reused 510 (from 1)
Receiving objects: 100% (510/510), 88.46 MiB | 7.21 MiB/s, done.
Resolving deltas: 100% (156/156), done.
❯ nvim pyproject.toml
❯ cat pyproject.toml
[project]
name = "test-sam2-with-uv-no-build-deps"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []
[project.optional-dependencies]
build = [
"torch==2.4.0+cu118",
"torchvision==0.19.0+cu118",
"setuptools>=70.0.0",
"wheel>=0.44.0",
]
# sam-2 = [
# "sam-2",
# "opencv-python>=4.10.0",
# ]
sam-2-demo = [
"jupyterlab>=4.1.6",
"matplotlib>=3.9.2",
]
[tool.uv]
extra-index-url = ["https://download.pytorch.org/whl/cu118"]
override-dependencies = ["tqdm>=4.64.1"]
no-build-isolation-package = ["sam-2"]
[tool.uv.sources]
sam-2 = { path = "segment-anything-2", editable = true }
❯ uv sync --extra build
Resolved 123 packages in 4.19s
Prepared 1 package in 1ms
Installed 26 packages in 255ms
+ filelock==3.13.1
+ fsspec==2024.2.0
+ jinja2==3.1.3
+ markupsafe==2.1.5
+ mpmath==1.3.0
+ networkx==3.2.1
+ numpy==1.26.3
+ nvidia-cublas-cu11==11.11.3.6
+ nvidia-cuda-cupti-cu11==11.8.87
+ nvidia-cuda-nvrtc-cu11==11.8.89
+ nvidia-cuda-runtime-cu11==11.8.89
+ nvidia-cudnn-cu11==9.1.0.70
+ nvidia-cufft-cu11==10.9.0.58
+ nvidia-curand-cu11==10.3.0.86
+ nvidia-cusolver-cu11==11.4.1.48
+ nvidia-cusparse-cu11==11.7.5.86
+ nvidia-nccl-cu11==2.20.5
+ nvidia-nvtx-cu11==11.8.86
+ pillow==10.2.0
+ setuptools==70.0.0
+ sympy==1.12
+ torch==2.4.0+cu118
+ torchvision==0.19.0+cu118
+ triton==3.0.0
+ typing-extensions==4.9.0
+ wheel==0.44.0
❯ nvim pyproject.toml
❯ cat pyproject.toml
[project]
name = "test-sam2-with-uv-no-build-deps"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []
[project.optional-dependencies]
build = [
"torch==2.4.0+cu118",
"torchvision==0.19.0+cu118",
"setuptools>=70.0.0",
"wheel>=0.44.0",
]
sam-2 = [
"sam-2",
"opencv-python>=4.10.0",
]
sam-2-demo = [
"jupyterlab>=4.1.6",
"matplotlib>=3.9.2",
]
[tool.uv]
extra-index-url = ["https://download.pytorch.org/whl/cu118"]
override-dependencies = ["tqdm>=4.64.1"]
no-build-isolation-package = ["sam-2"]
[tool.uv.sources]
sam-2 = { path = "segment-anything-2", editable = true }
❯ uv sync --extra sam-2 --extra sam-2-demo
Resolved 131 packages in 7.63s
Built sam-2 @ file:///home/hideakitai/develop/ml/sandbox/test_sam2_with_uv_no_build_deps/segment-anything-2
Prepared 93 packages in 1.95s
Uninstalled 1 package in 0.74ms
Installed 100 packages in 141ms
+ antlr4-python3-runtime==4.9.3
+ anyio==4.4.0
+ argon2-cffi==23.1.0
+ argon2-cffi-bindings==21.2.0
+ arrow==1.3.0
+ asttokens==2.4.1
+ async-lru==2.0.4
+ attrs==24.2.0
+ babel==2.16.0
+ beautifulsoup4==4.12.3
+ bleach==6.1.0
+ certifi==2022.12.7
+ cffi==1.17.0
+ charset-normalizer==2.1.1
+ comm==0.2.2
+ contourpy==1.3.0
+ cycler==0.12.1
+ debugpy==1.8.5
+ decorator==5.1.1
+ defusedxml==0.7.1
+ executing==2.1.0
+ fastjsonschema==2.20.0
+ fonttools==4.53.1
+ fqdn==1.5.1
+ h11==0.14.0
+ httpcore==1.0.5
+ httpx==0.27.2
+ hydra-core==1.3.2
+ idna==3.4
+ iopath==0.1.10
+ ipykernel==6.29.5
+ ipython==8.27.0
+ isoduration==20.11.0
+ jedi==0.19.1
+ json5==0.9.25
+ jsonpointer==3.0.0
+ jsonschema==4.23.0
+ jsonschema-specifications==2023.12.1
+ jupyter-client==8.6.2
+ jupyter-core==5.7.2
+ jupyter-events==0.10.0
+ jupyter-lsp==2.2.5
+ jupyter-server==2.14.2
+ jupyter-server-terminals==0.5.3
+ jupyterlab==4.1.6
+ jupyterlab-pygments==0.3.0
+ jupyterlab-server==2.24.0
+ kiwisolver==1.4.5
+ matplotlib==3.9.2
+ matplotlib-inline==0.1.7
+ mistune==3.0.2
+ nbclient==0.10.0
+ nbconvert==7.16.4
+ nbformat==5.10.4
+ nest-asyncio==1.6.0
+ notebook-shim==0.2.4
+ omegaconf==2.3.0
+ opencv-python==4.10.0.84
+ overrides==7.7.0
+ packaging==22.0
+ pandocfilters==1.5.1
+ parso==0.8.4
+ pexpect==4.9.0
+ platformdirs==4.2.2
+ portalocker==2.10.1
+ prometheus-client==0.20.0
+ prompt-toolkit==3.0.47
+ psutil==6.0.0
+ ptyprocess==0.7.0
+ pure-eval==0.2.3
+ pycparser==2.22
+ pygments==2.18.0
+ pyparsing==3.1.4
+ python-dateutil==2.9.0.post0
+ python-json-logger==2.0.7
+ pyyaml==6.0.2
+ pyzmq==26.2.0
+ referencing==0.35.1
+ requests==2.28.1
+ rfc3339-validator==0.1.4
+ rfc3986-validator==0.1.1
+ rpds-py==0.20.0
+ sam-2==1.0 (from file:///home/hideakitai/develop/ml/sandbox/test_sam2_with_uv_no_build_deps/segment-anything-2)
+ send2trash==1.8.3
+ six==1.16.0
+ sniffio==1.3.1
+ soupsieve==2.6
+ stack-data==0.6.3
+ terminado==0.18.1
+ tinycss2==1.3.0
+ tornado==6.4.1
+ tqdm==4.64.1
+ traitlets==5.14.3
+ types-python-dateutil==2.9.0.20240821
+ uri-template==1.3.0
+ urllib3==1.26.13
+ wcwidth==0.2.13
+ webcolors==24.8.0
+ webencodings==0.5.1
+ websocket-client==1.8.0
- wheel==0.44.0
❯ cd segment-anything-2/checkpoints/
❯ ./download_ckpts.sh
Downloading sam2_hiera_tiny.pt checkpoint...
--2024-09-04 01:01:52-- https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 3.164.121.107, 3.164.121.45, 3.164.121.115, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|3.164.121.107|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 155906050 (149M) [application/vnd.snesdev-page-table]
Saving to: ‘sam2_hiera_tiny.pt’
sam2_hiera_tiny.pt 100%[============================================>] 148.68M 31.5MB/s in 4.7s
2024-09-04 01:01:57 (31.8 MB/s) - ‘sam2_hiera_tiny.pt’ saved [155906050/155906050]
Downloading sam2_hiera_small.pt checkpoint...
--2024-09-04 01:01:57-- https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_small.pt
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 18.65.185.33, 18.65.185.117, 18.65.185.119, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|18.65.185.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 184309650 (176M) [application/vnd.snesdev-page-table]
Saving to: ‘sam2_hiera_small.pt’
sam2_hiera_small.pt 100%[============================================>] 175.77M 49.4MB/s in 3.9s
2024-09-04 01:02:01 (44.8 MB/s) - ‘sam2_hiera_small.pt’ saved [184309650/184309650]
Downloading sam2_hiera_base_plus.pt checkpoint...
--2024-09-04 01:02:01-- https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_base_plus.pt
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 18.65.185.33, 18.65.185.117, 18.65.185.119, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|18.65.185.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 323493298 (309M) [application/vnd.snesdev-page-table]
Saving to: ‘sam2_hiera_base_plus.pt’
sam2_hiera_base_plus.pt 100%[============================================>] 308.51M 39.5MB/s in 8.1s
2024-09-04 01:02:09 (38.0 MB/s) - ‘sam2_hiera_base_plus.pt’ saved [323493298/323493298]
Downloading sam2_hiera_large.pt checkpoint...
--2024-09-04 01:02:09-- https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 18.65.185.117, 18.65.185.119, 18.65.185.25, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|18.65.185.117|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 897952466 (856M) [application/vnd.snesdev-page-table]
Saving to: ‘sam2_hiera_large.pt’
sam2_hiera_large.pt 100%[============================================>] 856.35M 38.7MB/s in 25s
2024-09-04 01:02:33 (34.7 MB/s) - ‘sam2_hiera_large.pt’ saved [897952466/897952466]
All checkpoints are downloaded successfully.
❯ cd -
/home/hideakitai/develop/ml/sandbox/test_sam2_with_uv_no_build_deps
❯ code .
# Now I could run segment-anything-2/notebooks/ successfully! |
|
If anyone knows of a better way, please let me know 👏 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks for the great tool.
I have tried to make a segment-anything-2 and a PyTorch environment with uv to get the demo working.
However, as of 9/1/2024, PyTorch v2.4.0 requires
tqdm==4.64.1, and sam-2 (7e1596c) requirestqdm>=4.66.1. So, there is no way to resolve this dependency mismatch, but it will unlikely happen shortly.Still, I want to use uv to manage the project while avoiding dependency mismatch. What do you think is the best workaround? I tried to use
uv add --frozen, but if I use uv,uv syncwill be called somewhere, and the dependencies will not be preserved. I can get it to work by using Python and pip directly in the virtual environment without using uv, so is that the only way to do it?uv pip installresults in the same problem.I have attached a log to reproduce the problem. I would appreciate your wisdom.
The text was updated successfully, but these errors were encountered: