New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLIP to KerasCV #2331

Merged

divyashreepathihalli merged 42 commits into keras-team:master from divyashreepathihalli:CLIP_refactor

Feb 22, 2024

Collaborator

divyashreepathihalli commented Feb 2, 2024 •

edited

Loading

Thank you @DavidLandup0 for the initial code port (to tf.keras) for image encoder and pre-processor!


clip refactor

67b2796

innat reviewed

View reviewed changes

keras_cv/models/CLIP/clip_encoder.py Outdated Show resolved Hide resolved

innat reviewed

View reviewed changes

keras_cv/models/CLIP/clip_encoder.py Outdated Show resolved Hide resolved

innat reviewed

View reviewed changes

keras_cv/models/CLIP/clip_image_model.py Outdated Show resolved Hide resolved

innat reviewed

View reviewed changes

keras_cv/models/CLIP/clip_model.py Outdated Show resolved Hide resolved

innat reviewed

View reviewed changes

keras_cv/models/CLIP/clip_processor.py Outdated Show resolved Hide resolved

divyashreepathihalli marked this pull request as draft

February 2, 2024 04:44

Divyashree Sreepathihalli added 9 commits

February 2, 2024 18:52


code cleanup and reformat

88ae6a4


update encoder name

3aa5c6c


update clip encoder name

1f648b3


update clip encoder name in image encoder

3c4743d


add weights conversion script

54ec6e5


update setup to install keras-nlp

286d0c2


new black formatting

209e5da


add preset file

91e6ea9


update array

2219bc2

divyashreepathihalli requested a review from sampathweb

February 3, 2024 00:12

Divyashree Sreepathihalli and others added 13 commits

February 3, 2024 00:13


update clip prests kaggle handle

957b6c8


update text model

160d2a9


Merge branch 'keras-team:master' into clip_refactor_sub

3c391ed


update text encoder

681120c


update position embeddings

df73f23


update positonal embeddings

80bde9c


add attention masks

5f7b23b


update expanded mask

7530eed


revert previous commit

0211bd4


change causal masks

d488b75


undo previous commit

d9d1264


update attention masks

64d66b5


update clip encoder

de0be19

sampathweb reviewed

View reviewed changes

keras_cv/models/feature_extractor/clip/clip_image_model.py Outdated Show resolved Hide resolved

sampathweb reviewed

View reviewed changes

keras_cv/models/feature_extractor/clip/clip_image_model.py Show resolved Hide resolved

sampathweb reviewed

View reviewed changes

keras_cv/models/feature_extractor/clip/clip_model.py Outdated Show resolved Hide resolved

mattdangerw reviewed

View reviewed changes

Member

mattdangerw left a comment

Overall looks good! Looks like the tokenizer has a lot of dead code that could be cleaned up.

keras_cv/models/feature_extractor/clip/clip_encoder.py Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_encoder.py Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_encoder.py Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_encoder.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_encoder.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_model.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_tokenizer.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_tokenizer.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_tokenizer.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_tokenizer.py Outdated Show resolved Hide resolved

Divyashree Sreepathihalli added 2 commits

February 20, 2024 23:44


add modifications from review

38e00b7


change import checks

8eeb88e

divyashreepathihalli added the kokoro:force-run label

kokoro-team removed the kokoro:force-run label


update keras_nlp import check

d5b2534

divyashreepathihalli added the kokoro:force-run label

divyashreepathihalli requested review from sampathweb and mattdangerw

February 21, 2024 00:29

kokoro-team removed the kokoro:force-run label


update kokoro tests

9a66464

divyashreepathihalli added the kokoro:force-run label

kokoro-team removed the kokoro:force-run label

sampathweb reviewed

View reviewed changes

.kokoro/github/ubuntu/gpu/build.sh Show resolved Hide resolved

divyashreepathihalli requested a review from sampathweb

February 21, 2024 19:27

sampathweb approved these changes

View reviewed changes

Collaborator

sampathweb left a comment

LGTM. Lets wait for Matt to resolve his comments.


update kaggle preset version

a0b8e30

divyashreepathihalli added the kokoro:force-run label

kokoro-team removed the kokoro:force-run label

mattdangerw reviewed

View reviewed changes

keras_cv/models/feature_extractor/clip/clip_tokenizer.py Outdated Show resolved Hide resolved

keras_cv/models/feature_extractor/clip/clip_tokenizer.py Show resolved Hide resolved

mattdangerw approved these changes

View reviewed changes


update install instructions for keras-nlp

fe2ac12

divyashreepathihalli added the kokoro:force-run label

kokoro-team removed the kokoro:force-run label

divyashreepathihalli merged commit 15db57c into keras-team:master

10 checks passed

divyashreepathihalli added a commit that referenced this pull request


Revert "Add CLIP to KerasCV (#2331)"

9c098f1

This reverts commit 15db57c.

awsaf49 mentioned this pull request

CLIP result seems wrong #2387

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment