Installing detectron2 through pants

[johnathanchiu]

I am trying to install detectron2 via pants generate-lockfiles. I've added the following line to my requirements.txt:

detectron2 @ git+https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593

The error I get is

17:37:06.86 [INFO] Scheduler initialized.
17:37:07.68 [INFO] Completed: Generate lockfile for setuptools
17:37:08.43 [INFO] Completed: Generate lockfile for mypy
17:37:09.45 [INFO] Completed: Generate lockfile for python-default
17:37:09.45 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 77806 -> ~/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/d819101e76d1257235093025e8afc37bee57f317/bin/python -sE ~/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/d819101e76d1257235093025e8afc37bee57f317/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir ~/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache --log /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-y1SsV5/.tmp/pex-pip-log.vqga2q8u/pip.log download --dest /private/var/folders/wq/xvgpz9v17qsbhb1vwz087v5w0000gn/T/pants-sandbox-y1SsV5/.tmp/tmpbwvn219_/opt.homebrew.Cellar.python@3.12.3.12.4.Frameworks.Python.framework.Versions.3.12.bin.python3.12 Pillow==10.3.0 PyMuPDF==1.23.26 PyPDF2==3.0.1 SQLAlchemy==2.0.23 SQLAlchemy_Utils==0.41.1 absl_py==2.1.0 aiofiles==23.2.1 alembic==1.13.1 anyio==4.3.0 beautifulsoup4==4.12.3 boto3-stubs~=1.34.141 boto3~=1.34.141 botocore~=1.34.95 colorama~=0.4.6 datasets==2.14.6 detectron2@ git+https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593 docker==7.0.0 dspy_ai==2.4.12 fastapi==0.110.0 fire==0.5.0 google_api_python_client==2.134.0 google_auth==2.29.0 google_auth_oauthlib==1.0.0 httpx==0.27.0 instructor==1.3.4 llama-index==0.10.40 matplotlib==3.9.1 ml_collections==0.1.1 msal==1.28.0 networkx==3.3 nltk==3.8.1 numpy==1.26.4 omegaconf==2.3.0 openai==1.23.2 opencv-python==4.10.0.82 outlines==0.0.34 pandas-stubs pandas==2.2.2 pdfminer.six==20221105 pdfplumber==0.10.3 pex==2.2.2 playwright==1.41.2 protobuf==4.25.3 psycopg2_binary==2.9.9 pycocoevalcap==1.2 pydantic==2.7.4 pylibmagic==0.5.0 pypdfium2==4.30.0 pytesseract==0.3.10 pytest==8.2.2 pytest_mock==3.14.0 python-magic==0.4.27 python-multipart==0.0.9 pytorch_lightning==2.1.3 ragas==0.1.9 requests==2.31.0 rich==13.7.1 scikit_learn==1.5.0 setuptools==70.0.0 sqlmodel==0.0.18 starlette~=0.36.3 streamlit==1.35.0 structlog~=24.2.0 tenacity==8.2.3 tensorboard==2.16.2 torch==2.3.0 torchvision==0.18.0 tqdm==4.66.4 transformers==4.41.2 trio==0.24.0 types-Pillow==9.5.0 types-beautifulsoup4==4.12.0.20240511 types-colorama~=0.4.6 types-requests==2.31.0 types-tqdm==4.66.0.5 unstructured-client~=0.22.0 unstructured~=0.14.4 uvicorn==0.28.0 wandb==0.17.4 watchdog~=4.0.0 weaviate_client==4.6.5 websockets~=12.0.0 wheel~=0.43.0 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

[jsirois]

@johnathanchiu this project has bad packaging has very specific install instructions you didn't follow. It assumes it will only be installed in a venv that already has torch installed.

1st this is how you self-serve debug, stripping Pants and even Pex away:

# Make a fresh venv clean room:
:; python3.12 -mvenv debug.venv
:; debug.venv/bin/pip -q install -U pip
:; debug.venv/bin/pip --version
pip 24.1.2 from /home/jsirois/dev/pex-tool/pip/debug.venv/lib/python3.12/site-packages/pip (python 3.12)

# Now try to install the thing:
:; debug.venv/bin/pip install "detectron2 @ git+https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593"
Collecting detectron2@ git+https://github.com/facebookresearch/detectron2.git@2a420edb307c9bdf640f036d3b196bed474b8593
  Cloning https://github.com/facebookresearch/detectron2.git (to revision 2a420edb307c9bdf640f036d3b196bed474b8593) to /tmp/pip-install-p2tn8kc3/detectron2_6edf3acd472a495c83c88728fc54f5b0
  Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/detectron2.git /tmp/pip-install-p2tn8kc3/detectron2_6edf3acd472a495c83c88728fc54f5b0
  Running command git rev-parse -q --verify 'sha^2a420edb307c9bdf640f036d3b196bed474b8593'
  Running command git fetch -q https://github.com/facebookresearch/detectron2.git 2a420edb307c9bdf640f036d3b196bed474b8593
  Resolved https://github.com/facebookresearch/detectron2.git to commit 2a420edb307c9bdf640f036d3b196bed474b8593
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "/home/jsirois/dev/pex-tool/pip/debug.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/jsirois/dev/pex-tool/pip/debug.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/jsirois/dev/pex-tool/pip/debug.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-kiwg8s1c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-kiwg8s1c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-kiwg8s1c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-kiwg8s1c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 10, in <module>
      ModuleNotFoundError: No module named 'torch'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

So, from this clean room experiment, it appears the project's setup.py tries to import torch but there is no torch. Let's see:
https://github.com/facebookresearch/detectron2/blob/2a420edb307c9bdf640f036d3b196bed474b8593/setup.py#L10-L11

Ok... Well, you can import non-stdlib / non-setuptools stuff in a setup.py but only if you arrange for it to be present via PEP-518 build-system.requires. That just means slapping down a small pyproject.toml that they're missing in that project, like so:
jsirois/detectron2@e71aca9

And then:

:; debug.venv/bin/pip install "detectron2 @ git+https://github.com/jsirois/detectron2@e71aca907a4b8fdf976af26e47ee3343f86b3147"
Collecting detectron2@ git+https://github.com/jsirois/detectron2@e71aca907a4b8fdf976af26e47ee3343f86b3147
  Cloning https://github.com/jsirois/detectron2 (to revision e71aca907a4b8fdf976af26e47ee3343f86b3147) to /tmp/pip-install-m1yaltk0/detectron2_4b5670d30f8f49fbaa61af4af665ebd8
  Running command git clone --filter=blob:none --quiet https://github.com/jsirois/detectron2 /tmp/pip-install-m1yaltk0/detectron2_4b5670d30f8f49fbaa61af4af665ebd8
  Running command git rev-parse -q --verify 'sha^e71aca907a4b8fdf976af26e47ee3343f86b3147'
  Running command git fetch -q https://github.com/jsirois/detectron2 e71aca907a4b8fdf976af26e47ee3343f86b3147
  Running command git checkout -q e71aca907a4b8fdf976af26e47ee3343f86b3147
  Resolved https://github.com/jsirois/detectron2 to commit e71aca907a4b8fdf976af26e47ee3343f86b3147
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
...
Building wheels for collected packages: detectron2, fvcore, antlr4-python3-runtime
  Building wheel for detectron2 (pyproject.toml) ... done
  Created wheel for detectron2: filename=detectron2-0.6-cp312-cp312-linux_x86_64.whl size=5972440 sha256=3e24aaa61415a4e23ea6cc950d7093d6ab0082241826c7aad136e5a630e9bbc1
  Stored in directory: /home/jsirois/.cache/pip/wheels/98/a2/f6/f6014a2b0521d0dd908770f84dc19ce3272959e48db75ae2fa
  Building wheel for fvcore (pyproject.toml) ... done
  Created wheel for fvcore: filename=fvcore-0.1.5.post20221221-py3-none-any.whl size=61398 sha256=df1de2fe08cab8df8d4164a13d3a5b1f34a34ecdbb0dd4ef21321a6cd43236e4
  Stored in directory: /home/jsirois/.cache/pip/wheels/ed/9f/a5/e4f5b27454ccd4596bd8b62432c7d6b1ca9fa22aef9d70a16a
  Building wheel for antlr4-python3-runtime (pyproject.toml) ... done
  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.9.3-py3-none-any.whl size=144553 sha256=56a9ce77cd1cf8840a2a9935625dfa9df8d66d83242a77b71cc3d4227ee286be
  Stored in directory: /home/jsirois/.cache/pip/wheels/1f/be/48/13754633f1d08d1fbfc60d5e80ae1e5d7329500477685286cd
Successfully built detectron2 fvcore antlr4-python3-runtime
Installing collected packages: antlr4-python3-runtime, tqdm, termcolor, tensorboard-data-server, tabulate, six, setuptools, pyyaml, pyparsing, protobuf, portalocker, platformdirs, Pillow, pathspec, packaging, numpy, mypy-extensions, MarkupSafe, markdown, kiwisolver, grpcio, fonttools, cycler, cloudpickle, click, absl-py, yacs, werkzeug, python-dateutil, omegaconf, iopath, contourpy, black, tensorboard, matplotlib, hydra-core, fvcore, pycocotools, detectron2
Successfully installed MarkupSafe-2.1.5 Pillow-10.4.0 absl-py-2.1.0 antlr4-python3-runtime-4.9.3 black-24.4.2 click-8.1.7 cloudpickle-3.0.0 contourpy-1.2.1 cycler-0.12.1 detectron2-0.6 fonttools-4.53.1 fvcore-0.1.5.post20221221 grpcio-1.65.1 hydra-core-1.3.2 iopath-0.1.9 kiwisolver-1.4.5 markdown-3.6 matplotlib-3.9.1 mypy-extensions-1.0.0 numpy-2.0.1 omegaconf-2.3.0 packaging-24.1 pathspec-0.12.1 platformdirs-4.2.2 portalocker-2.10.1 protobuf-4.25.4 pycocotools-2.0.8 pyparsing-3.1.2 python-dateutil-2.9.0.post0 pyyaml-6.0.1 setuptools-71.1.0 six-1.16.0 tabulate-0.9.0 tensorboard-2.17.0 tensorboard-data-server-0.7.2 termcolor-2.4.0 tqdm-4.66.4 werkzeug-3.0.3 yacs-0.1.8

Not only was this extremely slow to compile, but it may not be the right compile either, because, IIUC you want to have the flavors (as well as versions) of torch and torchvision you need pre-installed before setup.py runs. That means dealing with the torch custom repos shenanigans to select amongst cpu vs CUDA X.Y, etc (I.E.: The 'Install PyTorch' matrix here: https://pytorch.org/). So you'll need to ensure the right index configuration is setup with Pants for all this to have a hope of plopping out the wheel you really want.

Stepping back, they do warn you about this! See bullet point 2: https://detectron2.readthedocs.io/en/latest/tutorials/install.html#requirements

The upshot - don't use the Git VCS URL, instead pre-build the wheel(s) you want per the instructions at https://detectron2.readthedocs.io/en/latest/tutorials/install.html and then host those pre-built wheels in your own find-links repository or index server or, of you only need wheels for Linux, use detectron2's flavor / version matrix to select yet more custom indexes: https://detectron2.readthedocs.io/en/latest/tutorials/install.html#install-pre-built-detectron2-linux-only. I know Pex and Pip and Python packaging insanity, but I don't know Pants any longer; so someone else would have to help you with configuring custom find-links / index servers with Pants, but you also should probably be able to figure that out yourself knowing you likely need this given Pants' docs out at https://pantsbuild.org.

Secondary thing to note - the debug steps I used above are generally useful and the trick of adding a pyproject.toml to an old setup.py project that does not declare its own requirements (or does via setup_requires - which is long deprecated and won't work) - is generally a good trick that can work for smaller projects than detectron2 that compile in reasonable amounts of time.

[jsirois]

I guess to generate pants lockfiles is there a way to add/install this dependency before hand?

Yes - that's what I demonstrated with my fork of detectron2 above (jsirois/detectron2@e71aca9). You can test using a VCS requirement pointing to that like I did in my example, but you probably want your own fork in the end once you've vetted this all works.

no need to mess with the specific "flavors".

Ok - above my pay grade except to say "works" may mean one hell of a lot slower when it's cpu vs. CUDA X.Y ... you know your target audience though.

[johnathanchiu]

Gotcha. I was hoping there was a work around for not needing my own fork for maintenance issues. My alternative approach, would be to write this into the BUILD file and then create a pyproject.toml which the python_requirements in the BUILD file point to that. Not sure if this would work because I'm not certain about the order of execution in this case.

What I am referring to is this: https://www.pantsbuild.org/2.21/docs/python/overview/third-party-dependencies#pep-621-compliant-pyprojecttoml. I'm still giving it a go. Let me know if my understanding of this is completely off.

EDIT: Upon re-reading, it seems like you can only specify one or the other. You cannot do both a requirements.txt and a pyproject.toml.

[jsirois]

And worse - your understanding is, in fact, completely off here. There is absolutely no way to get this to work without modifying (adding) the detectron2 pyproject.toml. You cannot do this with any other pyproject.toml. This is what I was getting at with my "The Python ecosystem has no standard for specifying an install recipe as a consumer of other projects. In other words, like: ..." example above.

[jsirois]

Gotcha. I was hoping there was a work around for not needing my own fork for maintenance issues.

There are exactly 2 paths to fix this only, you have no other choices:

1. Fork detectron2 to add build dependencies as I did in the worked example.
2. Instead, manually pre-build the wheels you need ahead of time and host these in a find-links repo or your own custom index and then wire that repo / index up to Pants.

For either 1 or 2 you'll need to do an out of band update step each time you wish to upgrade detecron2. In case 1, you re-establish the thin pyproject.toml at a newer commit. In case 2, you pre-build the wheels you need from that same newer commit. Case 1 is easier for you. Case 2 is easier for everyone but you. You take the time to build the wheels once and host them. All your developers and CI save a ton of time building the wheels afresh.

[jsirois]

I mean - you do have a 3rd choice actually, the recipe idea I mentioned that is not supported by the Python ecosystem, i.e.: "The Python ecosystem has no standard for specifying an install recipe as a consumer of other projects. In other words, like: ..."

You could ask Pants folks if they want to invent that. Pants certainly is flexible enough to handle such a recipe with its rule engine and custom rules. It would take some work though that Pants maintainers or you as a maybe willing new Pants contributor may or may not be willing to invest.

[johnathanchiu]

Thanks! I really appreciate the steps for debugging. That will come in handy! Let me go over this and I'll let you know if I have additional issues.