Import ONNX model zoo programs to iree_tests

[ScottTodd]

https://github.com/onnx/models, https://onnx.ai/models/

The iree_tests/onnx/ test suite currently imports "node" tests from onnx/backend/test/case/node. There are also a few "model" tests there onnx/backend/test/case/model, but many more models exist at https://github.com/onnx/models.

Import process:

• Fork https://github.com/nod-ai/SHARK-TestSuite/blob/main/iree_tests/onnx/import_tests.py
• For each model in the model zoo:
  • Determine some test inputs
  • Run through onnxruntime (?)
  • Record test outputs
  • Generate a test case in iree_tests
• Add test config files like https://github.com/nod-ai/SHARK-TestSuite/blob/main/iree_tests/configs/onnx_cpu_llvm_sync.json, going through import -> compile -> run for all test cases
• Prune the set of imported models as needed (e.g. dedup different opset versions)

[zjgarvey]

This sounds like a great idea. Can we add the zoo as a git submodule?

Can iree_tests do post-processing? If so, I don't really see the point in using anything else for public model testing (especially CI jobs).

[ScottTodd]

Can we add the zoo as a git submodule?

Yep, or it can be referenced from another local directory like

projects/
  SHARK-TestSuite/
  models/

Can iree_tests do post-processing? If so, I don't really see the point in using anything else for public model testing (especially CI jobs).

My view on post-processing is that the core IREE tools just operate on directly represented inputs and outputs. The compiler's job is to correctly translate an input program into an executable -- I really want to draw a hard line there if at all possible, or else the full mess of machine learning frameworks leaks down layers of the stack. A test generator that runs offline can do whatever pre- and post-processing it wants to produce those inputs and outputs. Where that gets tricky is when the tests can be considered "passed" with different numerical results. I'm not sure if lower precision (e.g. quantized) models running through the compiler and runtime are expected to produce results that are different beyond traditional per-element floating point comparison thresholds.

[zjgarvey]

My view on post-processing is that the core IREE tools just operate on directly represented inputs and outputs. The compiler's job is to correctly translate an input program into an executable -- I really want to draw a hard line there if at all possible, or else the full mess of machine learning frameworks leaks down layers of the stack. A test generator that runs offline can do whatever pre- and post-processing it wants to produce those inputs and outputs. Where that gets tricky is when the tests can be considered "passed" with different numerical results. I'm not sure if lower precision (e.g. quantized) models running through the compiler and runtime are expected to produce results that are different beyond traditional per-element floating point comparison thresholds.

That scope makes sense to me. I've been looking into issues with numerics mismatches in quantized models, and sometimes its because a node like onnx.GloabalAveragePool (conjugated by QDQ's) doesn't have a quantized implementation, and maybe we get a result 0.00001 below the rounding threshold for quantization and maybe onnxruntime gets 0.0001 above the rounding threshold. Now we are suddenly off by the scale after the output QDQ, and that can cause the numerics to diverge all over the place.

[ScottTodd]

Moving this issue to iree-test-suites: iree-org/iree-test-suites#6