Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLIP to KerasCV #2331

Merged
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
67b2796
clip refactor
divyashreepathihalli Feb 2, 2024
88ae6a4
code cleanup and reformat
Feb 2, 2024
3aa5c6c
update encoder name
Feb 2, 2024
1f648b3
update clip encoder name
Feb 2, 2024
3c4743d
update clip encoder name in image encoder
Feb 2, 2024
54ec6e5
add weights conversion script
Feb 2, 2024
286d0c2
update setup to install keras-nlp
Feb 2, 2024
209e5da
new black formatting
Feb 2, 2024
91e6ea9
add preset file
Feb 2, 2024
2219bc2
update array
Feb 3, 2024
957b6c8
update clip prests kaggle handle
Feb 3, 2024
160d2a9
update text model
Feb 7, 2024
3c391ed
Merge branch 'keras-team:master' into clip_refactor_sub
divyashreepathihalli Feb 7, 2024
681120c
update text encoder
Feb 8, 2024
df73f23
update position embeddings
Feb 8, 2024
80bde9c
update positonal embeddings
Feb 8, 2024
5f7b23b
add attention masks
Feb 8, 2024
7530eed
update expanded mask
Feb 8, 2024
0211bd4
revert previous commit
Feb 8, 2024
d488b75
change causal masks
Feb 8, 2024
d9d1264
undo previous commit
Feb 8, 2024
64d66b5
update attention masks
Feb 8, 2024
de0be19
update clip encoder
Feb 8, 2024
4b8c1ef
add print statements
Feb 9, 2024
54f02e8
update the pooler output
Feb 9, 2024
f831638
remove print statements
Feb 9, 2024
79de15d
Merge pull request #2 from divyashreepathihalli/clip_refactor_sub
divyashreepathihalli Feb 9, 2024
3868bb5
add tests and preset
Feb 9, 2024
719417e
Merge pull request #3 from divyashreepathihalli/clip_refactor_sub
divyashreepathihalli Feb 9, 2024
39ccb18
Merge branch 'keras-team:master' into CLIP_refactor
divyashreepathihalli Feb 9, 2024
95d9e10
cleanup and reformat
Feb 13, 2024
d4c7e16
update build
Feb 14, 2024
305fb0a
add copywrite to presets file
Feb 14, 2024
9e6ff3b
fix build state errors
Feb 14, 2024
1c88b7e
update github actions and add preprocessor test
Feb 16, 2024
eb2bd44
incorporate review comments
Feb 16, 2024
38e00b7
add modifications from review
Feb 20, 2024
8eeb88e
change import checks
Feb 21, 2024
d5b2534
update keras_nlp import check
Feb 21, 2024
9a66464
update kokoro tests
Feb 21, 2024
a0b8e30
update kaggle preset version
Feb 21, 2024
fe2ac12
update install instructions for keras-nlp
Feb 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/actions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ jobs:
pip install torch>=2.0.1+cpu
pip install "jax[cpu]"
pip install keras-core
pip install keras-nlp-nightly --no-deps
pip install tensorflow-text==2.15
divyashreepathihalli marked this conversation as resolved.
Show resolved Hide resolved
pip install -e ".[tests]" --progress-bar off --upgrade
- name: Test with pytest
env:
Expand Down Expand Up @@ -75,6 +77,7 @@ jobs:
run: |
pip install -r requirements.txt
pip install -e ".[tests]" --progress-bar off --upgrade
pip install keras-nlp-nightly
divyashreepathihalli marked this conversation as resolved.
Show resolved Hide resolved
- name: Test with pytest
env:
TEST_CUSTOM_OPS: false # TODO(ianstenbit): test custom ops, or figure out what our story is here
Expand Down
5 changes: 5 additions & 0 deletions .kokoro/github/ubuntu/gpu/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,21 +29,26 @@ then
pip install --extra-index-url https://download.pytorch.org/whl/cpu torch==2.1.0+cpu
pip install torchvision~=0.16.0
pip install "jax[cpu]"
pip install keras-nlp-nightly --no-deps
pip install tensorflow-text==2.15

elif [ "$KERAS_BACKEND" == "tensorflow" ]
then
echo "TensorFlow backend detected."
pip install -r requirements-tensorflow-cuda.txt --progress-bar off
pip install keras-nlp-nightly

elif [ "$KERAS_BACKEND" == "jax" ]
then
echo "JAX backend detected."
pip install -r requirements-jax-cuda.txt --progress-bar off
pip install keras-nlp-nightly

elif [ "$KERAS_BACKEND" == "torch" ]
then
echo "PyTorch backend detected."
pip install -r requirements-torch-cuda.txt --progress-bar off
pip install keras-nlp-nightly
divyashreepathihalli marked this conversation as resolved.
Show resolved Hide resolved
fi

pip install --no-deps -e "." --progress-bar off
Expand Down
1 change: 1 addition & 0 deletions keras_cv/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@
from keras_cv.models.backbones.vit_det.vit_det_aliases import ViTDetLBackbone
from keras_cv.models.backbones.vit_det.vit_det_backbone import ViTDetBackbone
from keras_cv.models.classification.image_classifier import ImageClassifier
from keras_cv.models.feature_extractor.clip import CLIP
from keras_cv.models.object_detection.retinanet.retinanet import RetinaNet
from keras_cv.models.object_detection.yolo_v8.yolo_v8_backbone import (
YOLOV8Backbone,
Expand Down
13 changes: 13 additions & 0 deletions keras_cv/models/feature_extractor/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright 2023 The KerasCV Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
23 changes: 23 additions & 0 deletions keras_cv/models/feature_extractor/clip/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Copyright 2023 The KerasCV Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from keras_cv.models.feature_extractor.clip.clip_image_model import (
CLIPImageEncoder,
)
from keras_cv.models.feature_extractor.clip.clip_model import CLIP
from keras_cv.models.feature_extractor.clip.clip_processor import CLIPProcessor
from keras_cv.models.feature_extractor.clip.clip_text_model import (
CLIPTextEncoder,
)
from keras_cv.models.feature_extractor.clip.clip_tokenizer import CLIPTokenizer
Loading
Loading