Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests for PyTorch models #63

Open
ScottTodd opened this issue Dec 18, 2024 · 1 comment
Open

Add tests for PyTorch models #63

ScottTodd opened this issue Dec 18, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@ScottTodd
Copy link
Member

Related issues:

I want to delete these old tests: https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests/pytorch/models, but want a replacement first. There are currently just two tests there, opt-125M and resnet50, each imported from "e2eshark" using this script. The e2eshark code exports using iree-turbine with the code in tools/stubs/turbinemodel.py and these test cases: resnet50/model.py, opt-125M/model.py.

I've started work on this on a branch: https://github.com/ScottTodd/iree-test-suites/tree/pytorch-models-setup. Some observations:

  • Pretrained PyTorch models are available from multiple sources. Most bottom out at nn.Module, but there doesn't appear to be a convenient way to reflect on an arbitrary nn.Module to get a canonical input signature (like with ONNX).

  • PyTorch models can be very flexible. It may be worth testing mutliple versions of each model: different data types, different implementations (e.g. attention op variants, different dynamic shape sizes, etc.)

  • iree-turbine may have some issues on Windows:

    tests/torchvision/resnet_test.py::test_resnet50 Windows fatal exception: access violation
    
    Current thread 0x00003e6c (most recent call first):
      File "D:\dev\projects\iree-test-suites\pytorch_models\.venv\Lib\site-packages\iree\turbine\aot\support\ir_utils.py", line 332 in create_tensor_global
      File "D:\dev\projects\iree-test-suites\pytorch_models\.venv\Lib\site-packages\iree\turbine\aot\support\procedural\globals.py", line 126 in track
      File "D:\dev\projects\iree-test-suites\pytorch_models\.venv\Lib\site-packages\iree\turbine\aot\support\procedural\exported_program.py", line 323 in _lift_tensor_to_global
      ...
    
  • Might first focus on export -> compile coverage, then later add runtime execution and comparison with native torch on CPU and other backends

@ScottTodd ScottTodd added the enhancement New feature or request label Dec 18, 2024
@ScottTodd ScottTodd self-assigned this Dec 18, 2024
@ScottTodd
Copy link
Member Author

  • iree-turbine may have some issues on Windows:

    tests/torchvision/resnet_test.py::test_resnet50 Windows fatal exception: access violation
    
    Current thread 0x00003e6c (most recent call first):
      File "D:\dev\projects\iree-test-suites\pytorch_models\.venv\Lib\site-packages\iree\turbine\aot\support\ir_utils.py", line 332 in create_tensor_global
      File "D:\dev\projects\iree-test-suites\pytorch_models\.venv\Lib\site-packages\iree\turbine\aot\support\procedural\globals.py", line 126 in track
      File "D:\dev\projects\iree-test-suites\pytorch_models\.venv\Lib\site-packages\iree\turbine\aot\support\procedural\exported_program.py", line 323 in _lift_tensor_to_global
      ...
    

FWIW, that test segfaults on Linux too, so some issues with my test setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant