Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run exports broken for packages built after split into libtorch and pytorch outputs #234

Closed
1 task done
carterbox opened this issue Apr 30, 2024 · 0 comments
Closed
1 task done

Run exports broken for packages built after split into libtorch and pytorch outputs #234

carterbox opened this issue Apr 30, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@carterbox
Copy link
Member

carterbox commented Apr 30, 2024 •
edited
Loading

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

The new output libtorch is great. Thanks for whomever put the effort into that. However, I ran into an issue when building against the newly split package.

build:
  number: 0
  skip: true  # [cuda_compiler_version == "None"]
  skip: true  # [cuda_compiler_version == "10.2"]
  skip: true  # [cuda_compiler_version == "11.0"]
  skip: true  # [cuda_compiler_version == "11.1"]
  # no pytorch on windows in conda-forge, see
  # https://github.com/conda-forge/pytorch-cpu-feedstock/issues/32
  skip: true  # [win]
  rpaths:
    - lib/
  # PyTorch libs are in site-packages intead of with other shared objects
    - {{ SP_DIR }}/torch/lib/

requirements:
  build:
    - {{ compiler('cuda') }}
    - {{ compiler('cxx') }}
  host:
    - cuda-version {{ cuda_compiler_version }}
  {% if cuda_major == 12 %}
    - cuda-cudart-dev
    - libcufft-dev
    - libcurand-dev
  {% endif %}
    - python
    - pip
    - pytorch
  # We must have the CUDA variant of pytorch
    - pytorch =*=cuda*
    - setuptools
    - setuptools-scm
    - toml
  run:
    - __cuda
    - python
    - pytorch =*=cuda*
    - scipy
    - numpy

I get these errors from conda-build about missing dependencies

   INFO (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): Needed DSO lib/libcufft.so.10 found in conda-forge/linux-64::cudatoolkit==11.2.2=hc23eb0c_13
  ERROR (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): Needed DSO lib/libc10.so found in ['conda-forge/linux-64::libtorch==2.1.2=cuda112_hf51c5c5_301']
  ERROR (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): .. but ['conda-forge/linux-64::libtorch==2.1.2=cuda112_hf51c5c5_301'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
  ERROR (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): Needed DSO lib/libtorch_cpu.so found in ['conda-forge/linux-64::libtorch==2.1.2=cuda112_hf51c5c5_301']
  ERROR (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): .. but ['conda-forge/linux-64::libtorch==2.1.2=cuda112_hf51c5c5_301'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): Needed DSO lib/python3.11/site-packages/torch/lib/libtorch_python.so found in conda-forge/linux-64::pytorch==2.1.2=cuda112_py311h9d54ea1_301
   INFO (carterbox-torch-radon,lib/python3.11/site-packages/torch_radon_cuda.cpython-311-x86_64-linux-gnu.so): Needed DSO lib/libcudart.so.11.0 found in conda-forge/linux-64::cudatoolkit==11.2.2=hc23eb0c_13

The cause of these errors is that my package links to libraries which are now separated into libtorch, but only direct dependencies (not higher-order dependencies) satisfy the link checker. The easy fix is that my package should now directly depend on libtorch. However, is it even possible to build a pytorch extension which links to libtorch_python.so and NOT libtorch_cpu.so/libc10.so? If this is not possible, then the pytorch package should also run_export libtorch since any python package linking to the pytorch libraries will need both pytorch and libtorch.

Installed packages

2024-04-30T14:29:09.9462823Z     _libgcc_mutex:     0.1-conda_forge                 conda-forge
2024-04-30T14:29:09.9463275Z     _openmp_mutex:     4.5-2_kmp_llvm                  conda-forge
2024-04-30T14:29:09.9463739Z     bzip2:             1.0.8-hd590300_5                conda-forge
2024-04-30T14:29:09.9464201Z     ca-certificates:   2024.2.2-hbcca054_0             conda-forge
2024-04-30T14:29:09.9464660Z     cuda-version:      11.2-hb11dac2_3                 conda-forge
2024-04-30T14:29:09.9465138Z     cudatoolkit:       11.2.2-hc23eb0c_13              conda-forge
2024-04-30T14:29:09.9465586Z     cudnn:             8.9.7.29-hbc23b4c_3             conda-forge
2024-04-30T14:29:09.9466052Z     filelock:          3.14.0-pyhd8ed1ab_0             conda-forge
2024-04-30T14:29:09.9466517Z     fsspec:            2024.3.1-pyhca7485f_0           conda-forge
2024-04-30T14:29:09.9466954Z     gmp:               6.3.0-h59595ed_1                conda-forge
2024-04-30T14:29:09.9467408Z     gmpy2:             2.1.5-py311he48d604_0           conda-forge
2024-04-30T14:29:09.9467862Z     icu:               73.2-h59595ed_0                 conda-forge
2024-04-30T14:29:09.9468305Z     jinja2:            3.1.3-pyhd8ed1ab_0              conda-forge
2024-04-30T14:29:09.9468764Z     ld_impl_linux-64:  2.40-h55db66e_0                 conda-forge
2024-04-30T14:29:09.9469288Z     libabseil:         20230802.1-cxx17_h59595ed_0     conda-forge
2024-04-30T14:29:09.9469760Z     libblas:           3.9.0-22_linux64_openblas       conda-forge
2024-04-30T14:29:09.9470224Z     libcblas:          3.9.0-22_linux64_openblas       conda-forge
2024-04-30T14:29:09.9470668Z     libexpat:          2.6.2-h59595ed_0                conda-forge
2024-04-30T14:29:09.9471382Z     libffi:            3.4.2-h7f98852_5                conda-forge
2024-04-30T14:29:09.9471837Z     libgcc-ng:         13.2.0-hc881cc4_6               conda-forge
2024-04-30T14:29:09.9472293Z     libgfortran-ng:    13.2.0-h69a702a_6               conda-forge
2024-04-30T14:29:09.9472761Z     libgfortran5:      13.2.0-h43f5ff8_6               conda-forge
2024-04-30T14:29:09.9473221Z     libhwloc:          2.10.0-default_h2fb2949_1000    conda-forge
2024-04-30T14:29:09.9473684Z     libiconv:          1.17-hd590300_2                 conda-forge
2024-04-30T14:29:09.9474141Z     liblapack:         3.9.0-22_linux64_openblas       conda-forge
2024-04-30T14:29:09.9474733Z     libmagma:          2.7.2-h09159a4_2                conda-forge
2024-04-30T14:29:09.9475195Z     libmagma_sparse:   2.7.2-h09159a4_3                conda-forge
2024-04-30T14:29:09.9475637Z     libnsl:            2.0.1-hd590300_0                conda-forge
2024-04-30T14:29:09.9476104Z     libopenblas:       0.3.27-pthreads_h413a1c8_0      conda-forge
2024-04-30T14:29:09.9476565Z     libprotobuf:       4.25.1-hf27288f_2               conda-forge
2024-04-30T14:29:09.9477008Z     libsqlite:         3.45.3-h2797004_0               conda-forge
2024-04-30T14:29:09.9477464Z     libstdcxx-ng:      13.2.0-h95c4c6d_6               conda-forge
2024-04-30T14:29:09.9477926Z     libtorch:          2.1.2-cuda112_hf51c5c5_301      conda-forge
2024-04-30T14:29:09.9478371Z     libuuid:           2.38.1-h0b41bf4_0               conda-forge
2024-04-30T14:29:09.9478833Z     libuv:             1.48.0-hd590300_0               conda-forge
2024-04-30T14:29:09.9479271Z     libxcrypt:         4.4.36-hd590300_1               conda-forge
2024-04-30T14:29:09.9479764Z     libxml2:           2.12.6-h232c23b_2               conda-forge
2024-04-30T14:29:09.9480216Z     libzlib:           1.2.13-hd590300_5               conda-forge
2024-04-30T14:29:09.9480660Z     llvm-openmp:       18.1.3-h4dfa4b3_0               conda-forge
2024-04-30T14:29:09.9481117Z     magma:             2.7.2-h2cf16e7_3                conda-forge
2024-04-30T14:29:09.9481589Z     markupsafe:        2.1.5-py311h459d7ec_0           conda-forge
2024-04-30T14:29:09.9482033Z     mkl:               2023.2.0-h84fe81f_50496         conda-forge
2024-04-30T14:29:09.9482485Z     mpc:               1.3.1-hfe3b2da_0                conda-forge
2024-04-30T14:29:09.9482927Z     mpfr:              4.2.1-h9458935_1                conda-forge
2024-04-30T14:29:09.9483383Z     mpmath:            1.3.0-pyhd8ed1ab_0              conda-forge
2024-04-30T14:29:09.9484332Z     nccl:              2.21.5.1-h0800d71_0             conda-forge
2024-04-30T14:29:09.9485326Z     ncurses:           6.4.20240210-h59595ed_0         conda-forge
2024-04-30T14:29:09.9485727Z     networkx:          3.3-pyhd8ed1ab_1                conda-forge
2024-04-30T14:29:09.9486110Z     numpy:             1.26.4-py311h64a7726_0          conda-forge
2024-04-30T14:29:09.9486473Z     openssl:           3.3.0-hd590300_0                conda-forge
2024-04-30T14:29:09.9486849Z     packaging:         24.0-pyhd8ed1ab_0               conda-forge
2024-04-30T14:29:09.9487202Z     pip:               24.0-pyhd8ed1ab_0               conda-forge
2024-04-30T14:29:09.9487579Z     python:            3.11.9-hb806964_0_cpython       conda-forge
2024-04-30T14:29:09.9487940Z     python_abi:        3.11-4_cp311                    conda-forge
2024-04-30T14:29:09.9488324Z     pytorch:           2.1.2-cuda112_py311h9d54ea1_301 conda-forge
2024-04-30T14:29:09.9488699Z     readline:          8.2-h8228510_1                  conda-forge
2024-04-30T14:29:09.9489060Z     setuptools:        69.5.1-pyhd8ed1ab_0             conda-forge
2024-04-30T14:29:09.9489449Z     setuptools-scm:    8.0.4-pyhd8ed1ab_1              conda-forge
2024-04-30T14:29:09.9489838Z     sleef:             3.5.1-h9b69904_2                conda-forge
2024-04-30T14:29:09.9490208Z     sympy:             1.12-pypyh9d50eac_103           conda-forge
2024-04-30T14:29:09.9490718Z     tbb:               2021.12.0-h00ab1b0_0            conda-forge
2024-04-30T14:29:09.9491096Z     tk:                8.6.13-noxft_h4845f30_101       conda-forge
2024-04-30T14:29:09.9491467Z     toml:              0.10.2-pyhd8ed1ab_0             conda-forge
2024-04-30T14:29:09.9491828Z     tomli:             2.0.1-pyhd8ed1ab_0              conda-forge
2024-04-30T14:29:09.9492205Z     typing-extensions: 4.11.0-hd8ed1ab_0               conda-forge
2024-04-30T14:29:09.9492581Z     typing_extensions: 4.11.0-pyha770c72_0             conda-forge
2024-04-30T14:29:09.9492962Z     tzdata:            2024a-h0c530f3_0                conda-forge
2024-04-30T14:29:09.9493313Z     wheel:             0.43.0-pyhd8ed1ab_1             conda-forge
2024-04-30T14:29:09.9493782Z     xz:                5.2.6-h166bdaf_0                conda-forge
2024-04-30T14:29:09.9494148Z     zstd:              1.5.5-hfc55251_0                conda-forge

Environment info

conda-forge ci
@carterbox carterbox added the bug Something isn't working label Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@carterbox