enable customize activation functions in clip vision encoder #1385
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1385
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit e4a9cb5 with merge base 9e65fa9 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
There was a problem hiding this comment.
Hey @Gasoonjia, thanks for opening this PR. This generally makes sense to me. Out of curiosity, what is your goal for changing the activation function? Do you intend to retrain the clip encoder from scratch?
| @@ -19,6 +19,7 @@ def clip_vision_encoder( | |||
| output_cls_projection: bool = False, | |||
| max_num_tiles: int = 4, | |||
| in_channels: int = 3, | |||
| hidden_act: torch.nn.Module = torch.nn.SiLU(), | |||
There was a problem hiding this comment.
nit: maybe call this intermediate_activation to be more clear which activation in the transformer it is changing?
| @@ -49,6 +50,7 @@ def clip_vision_encoder( | |||
| max_num_tiles (int): The maximum number of tiles that can be processed. This is used to | |||
| determine the size of the positional embeddings. | |||
| in_channels (int): The number of image input channels. | |||
| hidden_act (torch.nn.Module): The activation function used in the transformer layers. | |||
There was a problem hiding this comment.
| hidden_act (torch.nn.Module): The activation function used in the transformer layers. | |
| hidden_act (torch.nn.Module): The activation function used in the intermediate layers in the transformer encoder |
|
hey @Gasoonjia, you may be interested in our upcoming flamingo PR: https://github.com/pytorch/torchtune/pull/1357/files feel free to ping me in workplace https://fb.workplace.com/profile.php?id=61556984579937 |
|
Hey @RdoubleA thanks for comment! I've updated the PR and pls take a look! |
FYI, we had some work on llava transforms: #1057 |
|
Hey @felipemello1 glad to see you in the github! Yes I'm keeping my eye on your wonderful PR, and definetly need your help when I try to work on Flamingo! |
Thanks for sharing! Will try to leverage your work! |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1385 +/- ##
==========================================
+ Coverage 70.57% 72.81% +2.24%
==========================================
Files 272 272
Lines 12895 12895
==========================================
+ Hits 9101 9390 +289
+ Misses 3794 3505 -289 ☔ View full report in Codecov by Sentry. |
Context
What is the purpose of this PR? Is it to
Please link to any issues this PR addresses.
Changelog
What are the changes made in this PR?
This PR enables users to customize activation functions in clip vision encoder
Test plan
Please make sure to do each of the following if applicable to your PR. (If you're not sure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.)
pre-commit install)pytest testspytest tests -m integration_testUX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Example of docstring:
torchtune/torchtune/modules/vision_transformer.py
Line 285 in 6a7951f
Example in our docs: https://pytorch.org/torchtune/main/tutorials/qat_finetune.html#applying-qat-to-llama3-models