Fix Data2VecVision ONNX test #18587

ydshieh · 2022-08-11T16:03:14Z

What does this PR do?

Fix an issue from #18427. In short, Data2VecVision is for semantic segmentation.

Current CI test failure

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_048_data2vec_vision_image_segmentation
(line 412)  ValueError: Unrecognized configuration class <class 'transformers.models.data2vec.configuration_data2vec_vision.Data2VecVisionConfig'> for this kind of AutoModel: AutoModelForImageSegmentation.

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_048_data2vec_vision_image_segmentation
(line 412)  ValueError: Unrecognized configuration class <class 'transformers.models.data2vec.configuration_data2vec_vision.Data2VecVisionConfig'> for this kind of AutoModel: AutoModelForImageSegmentation.

HuggingFaceDocBuilderDev · 2022-08-11T16:30:32Z

The documentation is not available anymore as the PR was closed or merged.

src/transformers/onnx/config.py

ydshieh · 2022-08-12T13:48:23Z

With nn.AdaptiveAvgPool2d with output_size > 1, we get error

Current thread 0x00007f5bb924e740 (most recent call first):
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_io/saferepr.py", line 71 in repr_instance
  File "/usr/lib/python3.9/reprlib.py", line 62 in repr1
  File "/usr/lib/python3.9/reprlib.py", line 71 in <listcomp>
  File "/usr/lib/python3.9/reprlib.py", line 71 in _repr_iterable
  File "/usr/lib/python3.9/reprlib.py", line 78 in repr_tuple
  File "/usr/lib/python3.9/reprlib.py", line 60 in repr1
  File "/usr/lib/python3.9/reprlib.py", line 52 in repr
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_io/saferepr.py", line 60 in repr
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_io/saferepr.py", line 107 in saferepr
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_code/code.py", line 727 in repr_args
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_code/code.py", line 817 in repr_traceback_entry
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_code/code.py", line 867 in repr_traceback
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_code/code.py", line 926 in repr_excinfo
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/_code/code.py", line 666 in getrepr
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/nodes.py", line 475 in _repr_failure_py
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/python.py", line 1795 in repr_failure
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/reports.py", line 345 in from_item_and_call
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/runner.py", line 365 in pytest_runtest_makereport
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/main.py", line 322 in _main
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/yih_dar_huggingface_co/.local/lib/python3.9/site-packages/pytest/__main__.py", line 5 in <module>
  File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
  File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Segmentation fault

ydshieh · 2022-08-12T15:43:42Z

tests/onnx/test_onnx_v2.py

+
+        # It seems `nn.AdaptiveAvgPool2d` with `output_size` > 1 is not working with ONNX. We get `Segmentation fault`
+        # without any useful information.
+        if model.__class__.__name__ == "Data2VecVisionForSemanticSegmentation":


Maybe we should just skip this test for Data2VecVisionForSemanticSegmentation?

sgugger

Looks good to me, but should be approved by @michaelbenayoun or @lewtun before merging :-)

lewtun

Thanks for digging into this tricky bug @ydshieh !

I think semantic-segmentation can't be supported for this model, so suggest removing that feature altogether in the ONNX features list. That should remove the failing tests and we can revisit this once PyTorch adds support for the missing op

src/transformers/onnx/config.py

ydshieh · 2022-08-18T14:52:07Z

Thanks @lewtun for the review. Totally fine for me to remove semantic-segmentation as a supported feature. I will clean this PR a bit and pin you for final review then.

ydshieh · 2022-08-18T19:47:18Z

The failed tests are irrelevant. @lewtun it's ready for you to take a final look 🚀 !

FAILED tests/models/bigbird_pegasus/test_modeling_bigbird_pegasus.py::BigBirdPegasusStandaloneDecoderModelTest::test_sample_generate
FAILED tests/models/xlm_roberta_xl/test_modeling_xlm_roberta_xl.py::XLMRobertaXLModelTest::test_sample_generate_dict_output

lewtun

Thanks for iterating on this @ydshieh ! The change looks good to me and I've just rerun the tests to see if we can get a nice green CI before merging

lewtun · 2022-08-22T08:40:31Z

Now we have a different generation test failing:

FAILED tests/models/blenderbot/test_modeling_blenderbot.py::BlenderbotStandaloneDecoderModelTest::test_sample_generate
FAILED tests/models/xlm_roberta_xl/test_modeling_xlm_roberta_xl.py::XLMRobertaXLModelTest::test_sample_generate

Since this is unrelated to the current PR, is it OK to merge?

ydshieh · 2022-08-22T08:51:55Z

Hi @lewtun . Thank you for running. It is good to merge. But let me rebase and re-run it, as @gante fixed the issue in #18696 merged into main.

I will take care of the merge when everything is fine. Thanks again for the review.

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

ydshieh changed the title ~~fix~~ Fix Data2VecVision ONNX test Aug 11, 2022

ydshieh commented Aug 11, 2022

View reviewed changes

src/transformers/onnx/config.py Outdated Show resolved Hide resolved

ydshieh force-pushed the fix_data2vec_vision_onnx_test branch from b08a37d to 8e94061 Compare August 12, 2022 11:47

ydshieh marked this pull request as draft August 12, 2022 11:48

ydshieh force-pushed the fix_data2vec_vision_onnx_test branch from 8e94061 to ad0492b Compare August 12, 2022 12:00

ydshieh force-pushed the fix_data2vec_vision_onnx_test branch 3 times, most recently from a8b7838 to ea35838 Compare August 12, 2022 15:41

ydshieh commented Aug 12, 2022

View reviewed changes

ydshieh requested review from lewtun, michaelbenayoun and sgugger and removed request for lewtun and michaelbenayoun August 12, 2022 15:43

sgugger approved these changes Aug 12, 2022

View reviewed changes

ydshieh marked this pull request as ready for review August 12, 2022 15:53

ydshieh mentioned this pull request Aug 18, 2022

Rename second input dimension for ONNX-supported CV models #17976

Merged

5 tasks

lewtun reviewed Aug 18, 2022

View reviewed changes

src/transformers/onnx/config.py Outdated Show resolved Hide resolved

ydshieh force-pushed the fix_data2vec_vision_onnx_test branch from 979db82 to e05a27d Compare August 18, 2022 19:27

rachthree mentioned this pull request Aug 19, 2022

Determine framework automatically before ONNX export #18615

Merged

5 tasks

lewtun approved these changes Aug 22, 2022

View reviewed changes

lewtun mentioned this pull request Aug 22, 2022

Add SegFormer ONNX support #18006

Merged

ydshieh and others added 3 commits August 22, 2022 10:53

fix

7476324

Update src/transformers/onnx/config.py

36b7629

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

update

ed398a1

ydshieh force-pushed the fix_data2vec_vision_onnx_test branch from e05a27d to ed398a1 Compare August 22, 2022 08:53

ydshieh merged commit 3fa45db into main Aug 22, 2022

ydshieh deleted the fix_data2vec_vision_onnx_test branch August 22, 2022 09:28

oneraghavan pushed a commit to oneraghavan/transformers that referenced this pull request Sep 26, 2022

Fix Data2VecVision ONNX test (huggingface#18587)

a65f308

Co-authored-by: lewtun <lewis.c.tunstall@gmail.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Data2VecVision ONNX test #18587

Fix Data2VecVision ONNX test #18587

ydshieh commented Aug 11, 2022

HuggingFaceDocBuilderDev commented Aug 11, 2022 •

edited

Loading

ydshieh commented Aug 12, 2022 •

edited

Loading

ydshieh Aug 12, 2022

sgugger left a comment

lewtun left a comment

ydshieh commented Aug 18, 2022

ydshieh commented Aug 18, 2022

lewtun left a comment

lewtun commented Aug 22, 2022

ydshieh commented Aug 22, 2022

Fix Data2VecVision ONNX test #18587

Fix Data2VecVision ONNX test #18587

Conversation

ydshieh commented Aug 11, 2022

What does this PR do?

HuggingFaceDocBuilderDev commented Aug 11, 2022 • edited Loading

ydshieh commented Aug 12, 2022 • edited Loading

ydshieh Aug 12, 2022

Choose a reason for hiding this comment

sgugger left a comment

Choose a reason for hiding this comment

lewtun left a comment

Choose a reason for hiding this comment

ydshieh commented Aug 18, 2022

ydshieh commented Aug 18, 2022

lewtun left a comment

Choose a reason for hiding this comment

lewtun commented Aug 22, 2022

ydshieh commented Aug 22, 2022

HuggingFaceDocBuilderDev commented Aug 11, 2022 •

edited

Loading

ydshieh commented Aug 12, 2022 •

edited

Loading