Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tests for ONNX models #6

Open
ScottTodd opened this issue Aug 9, 2024 · 2 comments
Open

Add tests for ONNX models #6

ScottTodd opened this issue Aug 9, 2024 · 2 comments
Assignees
@ScottTodd
Labels
enhancement New feature or request

Comments

@ScottTodd
Copy link
Member

ScottTodd commented Aug 9, 2024 •
edited
Loading

See nod-ai/SHARK-TestSuite#275

Some work has gone into importing models already at https://github.com/nod-ai/SHARK-TestSuite/tree/main/e2eshark/onnx/models
@ScottTodd ScottTodd self-assigned this Sep 6, 2024
This was referenced Sep 6, 2024
Merged
Closed
ScottTodd added a commit that referenced this issue Sep 19, 2024
@ScottTodd
Begin testing models from the ONNX Model Zoo. (#23)
7b8bdf7 
Progress on #6.

A sample test report HTML file is available here:
https://scotttodd.github.io/iree-test-suites/onnx_models/report_2024_09_17.html

These new tests

* Download models from https://github.com/onnx/models
* Extract metadata from the models to determine which functions to call
with random data
* Run the models through [ONNX Runtime](https://onnxruntime.ai/) as a
reference implementation
* Import the models using `iree-import-onnx` (until we have a better
API: iree-org/iree#18289)
* Compile the models using `iree-compile` (currently just for `llvm-cpu`
but this could be parameterized later)
* Run the models using `iree-run-module`, checking outputs using
`--expected_output` and the reference data

Tests are written in Python using a set of pytest helper functions. As
the tests run, they can log details about what commands they are
running. When run locally, the `artifacts/` directory will contain all
the relevant files. More can be done in follow-up PRs to improve the
ergonomics there (like generating flagfiles).

Each test case can use XFAIL like
`@pytest.mark.xfail(raises=IreeRunException)`. As we test across
multiple backends or want to configure the test suite from another repo
(e.g. [iree-org/iree](https://github.com/iree-org/iree)), we can explore
more expressive marks.

Note that unlike the ONNX _operator_ tests, these tests use
`onnxruntime` and `iree-import-onnx` at test time. The operator tests
handle that as an infrequently ran offline step. We could do something
similar here, but the test inputs and outputs can be rather large for
real models and that gets into Git LFS or cloud storage territory.

If this test authoring model works well enough, we can do something
similar for other ML frameworks like TFLite
(#5).
@ScottTodd ScottTodd mentioned this issue Oct 16, 2024
Merged
ScottTodd added a commit to iree-org/iree that referenced this issue Oct 17, 2024
@ScottTodd
Run ONNX model tests as part of pkgci_test_onnx. (#18795)
8e54ed5 
Progress on iree-org/iree-test-suites#6.

Current tests included and their statuses:
```
PASSED onnx_models/tests/vision/classification_models_test.py::test_alexnet
PASSED onnx_models/tests/vision/classification_models_test.py::test_caffenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_densenet_121
PASSED onnx_models/tests/vision/classification_models_test.py::test_googlenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_inception_v2
PASSED onnx_models/tests/vision/classification_models_test.py::test_mnist
PASSED onnx_models/tests/vision/classification_models_test.py::test_resnet50_v1
PASSED onnx_models/tests/vision/classification_models_test.py::test_resnet50_v2
PASSED onnx_models/tests/vision/classification_models_test.py::test_shufflenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_shufflenet_v2
PASSED onnx_models/tests/vision/classification_models_test.py::test_squeezenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_vgg19
XFAIL onnx_models/tests/vision/classification_models_test.py::test_efficientnet_lite4
XFAIL onnx_models/tests/vision/classification_models_test.py::test_inception_v1
XFAIL onnx_models/tests/vision/classification_models_test.py::test_mobilenet
XFAIL onnx_models/tests/vision/classification_models_test.py::test_rcnn_ilsvrc13
XFAIL onnx_models/tests/vision/classification_models_test.py::test_zfnet_512
```

* CPU only for now. We haven't yet parameterized those tests to allow
for other backends or flags.
* Starting with `--override-ini=xfail_strict=false` so newly _passing_
tests won't fail the job. Newly _failing_ tests will fail the job. We
can add an external config file to customize which tests are expected to
fail like the onnx op tests if we want to track which are
passing/failing in this repository instead of in the test suite repo.

Sample logs:
https://github.com/iree-org/iree/actions/runs/11371239238/job/31633406729?pr=18795

ci-exactly: build_packages, test_onnx
@ScottTodd
Copy link
Member Author

An initial set of ONNX model tests has landed in https://github.com/iree-org/iree-test-suites/tree/main/onnx_models and is now running on every IREE PR / commit. Sample logs from a test run in IREE: https://github.com/iree-org/iree/actions/runs/11390747229/job/31693710855#step:8:19.

Test summary:

=========================== short test summary info ============================
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_alexnet
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_caffenet
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_densenet_121
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_googlenet
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_inception_v2
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_mnist
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_resnet50_v1
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_resnet50_v2
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_shufflenet
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_shufflenet_v2
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_squeezenet
PASSED iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_vgg19
XFAIL iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_efficientnet_lite4
XFAIL iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_inception_v1
XFAIL iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_mobilenet
XFAIL iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_rcnn_ilsvrc13
XFAIL iree-test-suites/onnx_models/tests/vision/classification_models_test.py::test_zfnet_512
================== 12 passed, 5 xfailed in 177.35s (0:02:57) ===================

Open tasks now:

@ScottTodd ScottTodd mentioned this issue Oct 17, 2024
Merged
ScottTodd added a commit to iree-org/iree that referenced this issue Oct 23, 2024
@ScottTodd
Document new external ONNX model and linalg operator test suites. (#1…
563b3e7 
…8819)

Progress on iree-org/iree-test-suites#2 and
iree-org/iree-test-suites#6 .
Eliasj42 pushed a commit to iree-org/iree that referenced this issue Oct 31, 2024
@ScottTodd
Document new external ONNX model and linalg operator test suites. (#1…
474a967 
…8819)

Progress on iree-org/iree-test-suites#2 and
iree-org/iree-test-suites#6 .

Signed-off-by: Elias Joseph <eljoseph@amd.com>
@ScottTodd ScottTodd added the enhancement New feature or request label Dec 11, 2024
@ScottTodd ScottTodd mentioned this issue Dec 18, 2024
Open
@ScottTodd
Copy link
Member Author

I'll try to import more tests and parameterize across backends today.

This was referenced Dec 18, 2024
Draft
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ScottTodd ScottTodd

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ScottTodd