Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Apple CoreML compilation tutorials for any SG model #1007

Closed
wants to merge 43 commits into from
Closed

Added Apple CoreML compilation tutorials for any SG model #1007

wants to merge 43 commits into from

Conversation

avideci
Copy link
Contributor

@avideci avideci commented May 15, 2023

CoreML Compilation Support For Any SG Model

I added a notebook for CoreML compilation, since it was requested by many of our users, and because it is a low-hanging fruit.

Convert to CoreML:

  1. Select any SG model (yolo-nas-s used by default), sing model.get(...)
  2. Convert to CoreML using the notebook - providing nn.Module and target checkpoint path for the CoreML model.

It uses torch tracing, which is recommended by apple as the go-to in terms of coreml source.
Note: The netron visualization does not work on colab, it assumes the notebook is running in localhost.

To compile you SD architecture or model to CoreML:

  1. Run the notebook on colab
  2. Download the file from /content/yolo-nas.mlmodel
  3. Drag to XCode
  4. Use in your iOS/OS X application!

The names or the inputs will always be input_1, input_2, and so on, to make it easy on researchers.
CoreML classes use keyword arguments for inference, so having the same input names helps a lot while working with different models and trying different things.

@dagshub
Copy link

dagshub bot commented May 15, 2023

Join the discussion on DagsHub!

@avideci
Copy link
Contributor Author

avideci commented May 15, 2023

Pre-Reviewed notebook in colab (where I ran all cells):
https://colab.research.google.com/drive/1vqa2_TJ3rgxbNaR5n--uVfpmkEkJjbrN?usp=sharing

@avideci
Copy link
Contributor Author

avideci commented May 15, 2023

Please note that is worked without calling prep_model_for_conversion,
We can add it if needed any maybe run regression on all the models, to see what breaks and what works.

@avideci avideci mentioned this pull request May 15, 2023
Closed
@shaydeci
Copy link
Contributor

Please note that is worked without calling prep_model_for_conversion,
We can add it if needed any maybe run regression on all the models, to see what breaks and what works.

Although it might still work without prep_model_for_conversion - we still need to call it, since this is where we fuse the repvgg branches (otherwise we will have decreased performance).

shaydeci
shaydeci suggested changes May 15, 2023
View reviewed changes
Copy link
Contributor

@shaydeci shaydeci left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the contribution (:
Some notes-

  • We need to call prep_model_for_conversion, otherwise we will not fuse the repvgg branches (especially important for YoloNAS).

  • If we take our pre-trained YoloNAS as an example, its best that the dummy input will be 640X640.

  • I think it wuold be great to add just some text before the code cells, that briefly explain what is being performed. Maybe even some of the text you added to this PR description.

@avideci avideci requested a review from shaydeci May 15, 2023 11:02
@avideci
Copy link
Contributor Author

avideci commented May 15, 2023

  • Added text cells
  • Added prep_for_conversoin
  • Changed to 640x640

@BloodAxe
Copy link
Contributor

Great work here! 💪
Maybe it make sense to move helper functions to
super_gradients.coreml namespace?

@avideci
Copy link
Contributor Author

avideci commented May 18, 2023

I added the code to super_gradients.models, just like the ONNX.
I also added tests for CoreML mlmodel / mlpackage.

I didn't want to abuse abstractions here, since we don't forecast many export functions as part of SG,
so I re-used some of the parameters (but kept their names).
Let me know if it makes sense. We can move the input preperation outside to a third function.

@avideci avideci requested review from shaydeci and removed request for shaydeci May 23, 2023 09:26
shaydeci
shaydeci approved these changes May 23, 2023
View reviewed changes
Copy link
Contributor

@shaydeci shaydeci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Louis-Dupont
Louis-Dupont approved these changes May 23, 2023
View reviewed changes
Copy link
Contributor

@Louis-Dupont Louis-Dupont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

avideci and others added 23 commits May 23, 2023 15:10
@avideci
Updated CoreML notebook with CR fixes
d037b17
@Louis-Dupont @avideci
Fix train_loader not initalized properly (#981)
5b7a589 
* fix

* fix typo

* typo
@lkdci @avideci
empty mask (#982)
8e06ce7
@shaydeci @avideci
changed requirement (#1002)
ae74b35
@BloodAxe @avideci
Fix convert_to_onnx to correctly handle case when input_shape is None…
b956f1c 
… and use input_size from prep_model_for_conversion_kwargs (#989)
@shaydeci @BloodAxe
@Louis-Dupont @avideci
Feature/sg 757 resume for spots (#870)
0f7f633 
* wip

* wip

* added functionality to get wandb latest ckpt before launch

* renamed param and simplified procedure

* added error for wandb

* added docs and example

* fix tests

---------

Co-authored-by: Eugene Khvedchenya <ekhvedchenya@gmail.com>
Co-authored-by: Louis-Dupont <35190946+Louis-Dupont@users.noreply.github.com>
@shaydeci @avideci
fixed prep_model_for_conversion call
67adb2d
@lkdci @avideci
Cityscapes AutoLabelling dataset (#1000)
4892bfd 
* CityscapesConcatDataset

* documentation

* ddrnet recipe

* unit test

* docs

* add to init
@Louis-Dupont @avideci
Predict on fused model (#998)
33bfebb 
* predict on fused model

* working version

* fix

* update

* update

* add benchmarl

* add reset

* torch.from_numpy

* fix
@Louis-Dupont @avideci
Proposal of issue template (#1018)
80c2b5e 
* first proposal

* improve

* minor change

* wip
@avideci
Added coreml exprot method to super_gradients.models, updated CoreML …
d97d795 
…export notebook
@avideci
Ran black
689289b
@avideci
Ran black
05a8c69
@avideci
Replaced exists with isfile
d6fb152
@Louis-Dupont @avideci
Fix doc (#1019)
95c04f4 
* replace []() with <a>

* fix

* fix

* fix

* rename tutorials
@T0T4R4 @avideci
Fixed missing encoding (Issue #999) (#1044)
e4f9885
@avideci
Remove input_shape and added input_size, added tests
92143e4
@avideci
Added tests, CR fixes
07d06e3
@BloodAxe @avideci
Cache model from platform locally (#1009)
851c761 
* Add caching of downloaded files
@avideci
Fixed dir test in mlpackage export
e77836c
@avideci
Formetted with black
cd5b8d7
@shaydeci @ranrubin @avideci
Feature/infra 000 nightly (#1051)
49d89d9 
* add nightly build support

* add nightly build support sanity

* added nightly param to skip md files

* add nightly build support sanity

* add nightly build support sanity

---------

Co-authored-by: ranrubin <ranrubin@gmail.com>
@avideci
Merge branch 'feature/SG-000_add_coreml_compilation_notebook' of gith…
e14634c 
…ub.com:Deci-AI/super-gradients into feature/SG-000_add_coreml_compilation_notebook
@avideci
Copy link
Contributor Author

avideci commented May 23, 2023

Squashed and signed in 62936fc

@avideci avideci closed this May 23, 2023
@avideci avideci mentioned this pull request May 23, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BloodAxe BloodAxe BloodAxe left review comments

@shaydeci shaydeci shaydeci approved these changes

@Louis-Dupont Louis-Dupont Louis-Dupont approved these changes

@ofrimasad ofrimasad ofrimasad approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@avideci @shaydeci @BloodAxe @ofrimasad @Louis-Dupont @lkdci @T0T4R4