Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTX export for visual prompting task #2274

Merged
merged 57 commits into from
Jul 7, 2023

Conversation

sungchul2
Copy link
Contributor

@sungchul2 sungchul2 commented Jun 28, 2023

Summary

This PR includes:

  • enable otx export
    • export dividing SAM into image encoder and decoder (prompt encoder + mask decoder)
  • enable otx eval with OpenVINO IR
  • refactoring & bug fix

I followed a flow of OpenVINOSegmentationTask for OpenVINOVisualPromptingTask, so they look so similar.
But, for visual prompting task, there are two models, image encoder and decoder.
I think this point is just difference between both tasks.

TODOs

  • otx export
    • ONNX
      image
    • IR
  • OpenVINO inference
    • sync inference (as default)
      $ otx eval
      2023-07-06 10:52:00,838 | INFO : Avg time per image: 4.619890505930892 secs
      2023-07-06 10:52:00,838 | INFO : Total time: 124.73704366013408 secs
      2023-07-06 10:52:00,838 | INFO : Visual Prompting OpenVINO inference completed
      2023-07-06 10:52:00,839 | INFO : Computing mDice
      2023-07-06 10:52:03,808 | INFO : mDice after evaluation: 0.9487857719406433
    • async inference
      • It showed performance degradation, need to check
  • Unit test
  • Integration test
  • rebase

How to test

Unit test (local)

$ tox -vv -e unittest-all-py310 -- tests/unit/algorithms/visual_prompting
====================== 165 passed, 110 warnings in 33.36s ======================

Integration test (local)

$ tox -vv -e tests-all-py310 -- tests/integration/cli/visual_prompting/test_visual_prompting.py
=================== 8 passed, 1 warning in 122.65s (0:02:02) ===================

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@github-actions github-actions bot added ALGO Any changes in OTX Algo Tasks implementation API Any changes in OTX API CLI Any changes in OTE CLI DOC Improvements or additions to documentation TEST Any changes in tests labels Jun 28, 2023
@sungchul2 sungchul2 force-pushed the visual-prompting-export branch 3 times, most recently from b6d22b6 to 4b22336 Compare June 30, 2023 14:58
@sungchul2 sungchul2 added this to the 1.4.0 milestone Jun 30, 2023
@sungchul2 sungchul2 force-pushed the visual-prompting-export branch 2 times, most recently from 24b6b62 to 469279e Compare July 4, 2023 00:41
@github-actions github-actions bot removed API Any changes in OTX API DOC Improvements or additions to documentation CLI Any changes in OTE CLI ALGO Any changes in OTX Algo Tasks implementation labels Jul 4, 2023
@sungchul2 sungchul2 force-pushed the visual-prompting-export branch from 34b9817 to 6e803a5 Compare July 6, 2023 08:34
@github-actions github-actions bot added API Any changes in OTX API CLI Any changes in OTE CLI ALGO Any changes in OTX Algo Tasks implementation DOC Improvements or additions to documentation labels Jul 6, 2023
@sungchul2 sungchul2 marked this pull request as ready for review July 6, 2023 08:36
@sungchul2 sungchul2 requested a review from a team as a code owner July 6, 2023 08:36
@sungchul2 sungchul2 requested a review from wonjuleee July 6, 2023 08:49
@sungchul2 sungchul2 enabled auto-merge (squash) July 7, 2023 01:50
@sungchul2 sungchul2 requested a review from eunwoosh July 7, 2023 01:52
Copy link
Contributor

@jaegukhyun jaegukhyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments. Frankly there are many codes that is not fully understood.
I think we should revisit this after FF. Before merging this PR I want to ensure there's no big accuracy drop before and after export. Do you have some data for this? I want to make sure of this since preprocessing and postprocessing of OV model is implemented in this PR, not from ModelAPI.

@sungchul2
Copy link
Contributor Author

I left some comments. Frankly there are many codes that is not fully understood. I think we should revisit this after FF. Before merging this PR I want to ensure there's no big accuracy drop before and after export. Do you have some data for this? I want to make sure of this since preprocessing and postprocessing of OV model is implemented in this PR, not from ModelAPI.

From this version, exported model showed slight mDice performance drop from 0.9503 to 0.9488.
As you said, it may be due to not using modules from modelapi.
I'm proceeding to change preprocessing and postprocessing to using modelapi in the next PR (#2311).

@sungchul2 sungchul2 merged commit bad2081 into openvinotoolkit:develop Jul 7, 2023
@sungchul2 sungchul2 deleted the visual-prompting-export branch July 7, 2023 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ALGO Any changes in OTX Algo Tasks implementation API Any changes in OTX API CLI Any changes in OTE CLI DOC Improvements or additions to documentation TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants