Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run ONNX model tests as part of pkgci_test_onnx. #18795

Merged
merged 2 commits into from
Oct 17, 2024

Conversation

ScottTodd
Copy link
Member

@ScottTodd ScottTodd commented Oct 16, 2024

Progress on iree-org/iree-test-suites#6.

Current tests included and their statuses:

PASSED onnx_models/tests/vision/classification_models_test.py::test_alexnet
PASSED onnx_models/tests/vision/classification_models_test.py::test_caffenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_densenet_121
PASSED onnx_models/tests/vision/classification_models_test.py::test_googlenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_inception_v2
PASSED onnx_models/tests/vision/classification_models_test.py::test_mnist
PASSED onnx_models/tests/vision/classification_models_test.py::test_resnet50_v1
PASSED onnx_models/tests/vision/classification_models_test.py::test_resnet50_v2
PASSED onnx_models/tests/vision/classification_models_test.py::test_shufflenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_shufflenet_v2
PASSED onnx_models/tests/vision/classification_models_test.py::test_squeezenet
PASSED onnx_models/tests/vision/classification_models_test.py::test_vgg19
XFAIL onnx_models/tests/vision/classification_models_test.py::test_efficientnet_lite4
XFAIL onnx_models/tests/vision/classification_models_test.py::test_inception_v1
XFAIL onnx_models/tests/vision/classification_models_test.py::test_mobilenet
XFAIL onnx_models/tests/vision/classification_models_test.py::test_rcnn_ilsvrc13
XFAIL onnx_models/tests/vision/classification_models_test.py::test_zfnet_512
  • CPU only for now. We haven't yet parameterized those tests to allow for other backends or flags.
  • Starting with --override-ini=xfail_strict=false so newly passing tests won't fail the job. Newly failing tests will fail the job. We can add an external config file to customize which tests are expected to fail like the onnx op tests if we want to track which are passing/failing in this repository instead of in the test suite repo.

Sample logs: https://github.com/iree-org/iree/actions/runs/11371239238/job/31633406729?pr=18795

ci-exactly: build_packages, test_onnx

@ScottTodd ScottTodd added infrastructure Relating to build systems, CI, or testing integrations/onnx ONNX integration work labels Oct 16, 2024
@ScottTodd ScottTodd marked this pull request as ready for review October 16, 2024 19:16
Copy link
Collaborator

@saienduri saienduri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, yeah would be nice to have a way to control the xfail configs here in iree itself

@ScottTodd ScottTodd merged commit 8e54ed5 into iree-org:main Oct 17, 2024
27 checks passed
@ScottTodd ScottTodd deleted the onnx-model-tests branch October 17, 2024 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Relating to build systems, CI, or testing integrations/onnx ONNX integration work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants