trying to build torch_migraphx I get errors. I got those solved (see below) but running tests mostly fail. #129

differentprogramming · 2024-06-06T07:02:41Z

Problem Description

I tried to build torch_migraphx, but got an error.
Note that even though it says that it can't find the torch module, in any python code I write it does import as 2.4.0.dev20240604+rocm6.1

Note that the form didn't let me choose my actual card which is a Radeon Instinct MI50 32 gb which reports as

  Name:                    gfx906                             
  Marketing Name:          AMD Radeon Graphics                
      Name:                    amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-

Error output:

Defaulting to user installation because normal site-packages is not writeable
Processing /home/user/torch_migraphx/py
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
          self.run_setup()
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 46, in <module>
          import torch
      ModuleNotFoundError: No module named 'torch'
      [end of output]


### Operating System

OS: NAME="Ubuntu" VERSION="22.04.4 LTS (Jammy Jellyfish)"

### CPU

Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz

### GPU

AMD Radeon Pro VII

### ROCm Version

ROCm 6.1.0

### ROCm Component

_No response_

### Steps to Reproduce

As it says in your README.me

git clone https://github.com/ROCmSoftwarePlatform/torch_migraphx.git
cd ./torch_migraphx/py
export TORCH_CMAKE_PATH=$(python -c "import torch; print(torch.utils.cmake_prefix_path)")
pip install .

### (Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module version 6.7.0 is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.13
Runtime Ext Version:     1.4
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
  Uuid:                    CPU-XX                             
  Marketing Name:          Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   4500                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            12                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    131573080(0x7d7a558) KB            
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    131573080(0x7d7a558) KB            
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    131573080(0x7d7a558) KB            
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx906                             
  Uuid:                    GPU-528858c173c6451d               
  Marketing Name:          AMD Radeon Graphics                
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      8192(0x2000) KB                    
  Chip ID:                 26273(0x66a1)                      
  ASIC Revision:           1(0x1)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   1800                               
  BDFID:                   26368                              
  Internal Node ID:        1                                  
  Compute Unit:            64                                 
  SIMDs per CU:            4                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   1                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          64(0x40)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        40(0x28)                           
  Max Work-item Per CU:    2560(0xa00)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 471                                
  SDMA engine uCode::      145                                
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    33538048(0x1ffc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    33538048(0x1ffc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Recommended Granule:0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***

Additional Information

I installed the drivers with apt get-install, nothing custom or fancy.
And I got pytorch from the nightly because that's the only place where you can get support for ROCm 6.1
And I got ONNX from /rocm/manylinux/rocm-rel-6.1/ even though the documentation still points at 6.0 libraries instead

The text was updated successfully, but these errors were encountered:

differentprogramming · 2024-06-06T07:11:20Z

I think I solved it.
I noticed that TORCH_CMAKE_PATH isn't in my environment even though the script should set it, so I set it by hand.
Now I get a different error: "CMake must be installed..."
I'll install cmake and try again.

differentprogramming · 2024-06-06T07:19:33Z

After installing cmake with sudo apt-get install cmake I get a different failure.

Defaulting to user installation because normal site-packages is not writeable
Processing /home/user/torch_migraphx/py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting tabulate
  Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Requirement already satisfied: numpy>=1.20.0 in /home/user/.local/lib/python3.10/site-packages (from torch-migraphx==0.0.3) (1.26.4)
Requirement already satisfied: packaging in /home/user/.local/lib/python3.10/site-packages (from torch-migraphx==0.0.3) (24.0)
Collecting pybind11-global
  Using cached pybind11_global-2.12.0-py3-none-any.whl (425 kB)
Requirement already satisfied: torch>=1.11.0 in /home/user/.local/lib/python3.10/site-packages (from torch-migraphx==0.0.3) (2.4.0.dev20240604+rocm6.1)
Requirement already satisfied: filelock in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (3.13.1)
Requirement already satisfied: jinja2 in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (3.1.3)
Requirement already satisfied: sympy in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (1.12)
Requirement already satisfied: fsspec in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (2024.2.0)
Requirement already satisfied: networkx in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (3.2.1)
Requirement already satisfied: typing-extensions>=4.8.0 in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (4.8.0)
Requirement already satisfied: pytorch-triton-rocm==3.0.0+bbe6246e37 in /home/user/.local/lib/python3.10/site-packages (from torch>=1.11.0->torch-migraphx==0.0.3) (3.0.0+bbe6246e37)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/lib/python3/dist-packages (from jinja2->torch>=1.11.0->torch-migraphx==0.0.3) (2.0.1)
Requirement already satisfied: mpmath>=0.19 in /home/user/.local/lib/python3.10/site-packages (from sympy->torch>=1.11.0->torch-migraphx==0.0.3) (1.2.1)
Building wheels for collected packages: torch-migraphx
  Building wheel for torch-migraphx (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for torch-migraphx (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [88 lines of output]
      /usr/lib/python3/dist-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running bdist_wheel
      running build
      running build_py
      running egg_info
      writing manifest file 'torch_migraphx.egg-info/SOURCES.txt'
      running build_ext
      -- The C compiler identification is GNU 11.4.0
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Looking for pthread.h
      -- Looking for pthread.h - found
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      CMake Error at CMakeLists.txt:12 (find_package):
        By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
        asked CMake to find a package configuration file provided by "pybind11",
        but CMake did not find one.
      
        Could not find a package configuration file provided by "pybind11" with any
        of the following names:
      
          pybind11Config.cmake
          pybind11-config.cmake
      
        Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
        "pybind11_DIR" to a directory containing one of the above files.  If
        "pybind11" provides a separate development package or SDK, be sure it has
        been installed.
      
      
      -- Configuring incomplete, errors occurred!
      See also "/home/user/torch_migraphx/py/build/temp.linux-x86_64-3.10/CMakeFiles/CMakeOutput.log".
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 230, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
          self.run_setup()
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 114, in <module>
          setup(
        File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "setup.py", line 71, in run
          self.build_extension(ext)
        File "setup.py", line 103, in build_extension
          subprocess.check_call(['cmake', ext.sourcedir] + cmake_args,
        File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/home/user/torch_migraphx/py', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/user/torch_migraphx/py/build/lib.linux-x86_64-3.10', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DCMAKE_PREFIX_PATH=/home/user/.local/lib/python3.10/site-packages/torch/share/cmake', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
      [end of output]

differentprogramming · 2024-06-06T07:25:52Z

I tried "pip install pybind11" and while that installed something, that did not change the errors.

differentprogramming · 2024-06-06T07:38:38Z

I uninstalled from pip and tried to build pybind11 directly from https://github.com/pybind/pybind11

This reported: Successfully installed pybind11-2.13.0.dev1

This did not change the error.

This seems like a common problem here
pybind/pybind11#1379
and here onnx/onnx#1624

Since I'm not used to these packages, I have no idea how to undo whatever running "pip install ." did in the pybind11 git clone, nor how to fix the installation!

differentprogramming · 2024-06-06T07:53:22Z

Finally solve it with:
pip install "pybind11[global]"

SO all of the steps I needed to get it to build were:

set TORCH_CMAKE_PATH manually to /home/user/.local/lib/python3.10/site-packages/torch/share/cmake which is what python -c "import torch; print(torch.utils.cmake_prefix_path) returned
install cmake with sudo apt-get install cmake
run pip install "pybind11[global]"

differentprogramming · 2024-06-06T08:09:39Z

Now that it built I tested it. Most tests failed.
I ran pytest ./torch_migraphx/tests
The final result was
= 502 failed, 82 passed, 8 skipped, 2 xfailed, 345 warnings in 125.69s (0:02:05) =
I've uploaded the results.
Note that pytorch tests seem to work and cuda is available in pytorch through rocm
Note also that MIGraphX test works and ends with

Batch size: 1
Rate: 115.191 inferences/sec
Total time: 8.68127ms
Total instructions time: 11.1258ms
Overhead time: 0.0482709ms, -2.44455ms
Overhead: 1%, -28%
[ MIGraphX Version: 2.9.0.c38a587ed ] Complete: ./migraphx-driver perf --model resnet50

output.txt

james-banks · 2024-06-30T00:48:39Z

Now that it built I tested it. Most tests failed. I ran pytest ./torch_migraphx/tests The final result was = 502 failed, 82 passed, 8 skipped, 2 xfailed, 345 warnings in 125.69s (0:02:05) = I've uploaded the results. Note that pytorch tests seem to work and cuda is available in pytorch through rocm Note also that MIGraphX test works and ends with
Batch size: 1
Rate: 115.191 inferences/sec
Total time: 8.68127ms
Total instructions time: 11.1258ms
Overhead time: 0.0482709ms, -2.44455ms
Overhead: 1%, -28%
[ MIGraphX Version: 2.9.0.c38a587ed ] Complete: ./migraphx-driver perf --model resnet50
output.txt

Thanks to your help, I managed to get it installed past a lot of these errors too.

Not much improvement on the tests:

548 failed, 87 passed, 6 skipped, 2 xfailed, 365 warnings in 81.64s (0:01:21) =

differentprogramming changed the title ~~trying to build torch_migraphx I get "no module named 'torch'" even though when I check torch.__version__ is 2.4.0.dev20240604+rocm6.1~~ trying to build torch_migraphx I get errors. I keep solving problems then having more problems so far. Jun 6, 2024

differentprogramming changed the title ~~trying to build torch_migraphx I get errors. I keep solving problems then having more problems so far.~~ trying to build torch_migraphx I get errors. I got those solved (see below) but running tests mostly fail. Jun 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trying to build torch_migraphx I get errors. I got those solved (see below) but running tests mostly fail. #129

trying to build torch_migraphx I get errors. I got those solved (see below) but running tests mostly fail. #129

differentprogramming commented Jun 6, 2024 •

edited

Loading

differentprogramming commented Jun 6, 2024

differentprogramming commented Jun 6, 2024 •

edited

Loading

differentprogramming commented Jun 6, 2024

differentprogramming commented Jun 6, 2024 •

edited

Loading

differentprogramming commented Jun 6, 2024

differentprogramming commented Jun 6, 2024 •

edited

Loading

james-banks commented Jun 30, 2024

trying to build torch_migraphx I get errors. I got those solved (see below) but running tests mostly fail. #129

trying to build torch_migraphx I get errors. I got those solved (see below) but running tests mostly fail. #129

Comments

differentprogramming commented Jun 6, 2024 • edited Loading

Problem Description

Additional Information

differentprogramming commented Jun 6, 2024

differentprogramming commented Jun 6, 2024 • edited Loading

differentprogramming commented Jun 6, 2024

differentprogramming commented Jun 6, 2024 • edited Loading

differentprogramming commented Jun 6, 2024

differentprogramming commented Jun 6, 2024 • edited Loading

james-banks commented Jun 30, 2024

differentprogramming commented Jun 6, 2024 •

edited

Loading

differentprogramming commented Jun 6, 2024 •

edited

Loading

differentprogramming commented Jun 6, 2024 •

edited

Loading

differentprogramming commented Jun 6, 2024 •

edited

Loading