Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI - Add kaggle creds to pull model #1459

Merged
merged 11 commits into from
Mar 4, 2024
Merged

CI - Add kaggle creds to pull model #1459

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bc97548
CI - Add kaggle creds to pull model
sampathweb Feb 22, 2024
452502f
add kaggle env variables
sampathweb Feb 23, 2024
252ed7f
Kaggle env:
sampathweb Feb 23, 2024
8c4e5a5
Kaggle env:
sampathweb Feb 23, 2024
5075745
Kaggle env:
sampathweb Feb 23, 2024
3b734e8
Kaggle env:
sampathweb Feb 23, 2024
1e77380
Update Build script for Kokoro
sampathweb Feb 26, 2024
d49329a
Add Kaggle env var
sampathweb Feb 26, 2024
71a2e75
set gemma preset to extra_large
sampathweb Mar 1, 2024
2f426ed
Change Gemma small preset to bfloat16
sampathweb Mar 4, 2024
0842f70
Change Gemma small preset to xlarge
sampathweb Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .kokoro/github/ubuntu/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
set -e

export KAGGLE_KEY="$(cat ${KOKORO_KEYSTORE_DIR}/73361_keras_kaggle_secret_key)"
export KAGGLE_USERNAME="$(cat ${KOKORO_KEYSTORE_DIR}/73361_keras_kaggle_username)"

if [[ -z "${KAGGLE_KEY}" ]]; then
echo "KAGGLE_KEY is NOT set"
exit 1
fi

if [[ -z "${KAGGLE_USERNAME}" ]]; then
echo "KAGGLE_USERNAME is NOT set"
exit 1
fi

set -x

cd "${KOKORO_ROOT}/"
Expand Down
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/jax/continuous.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "jax"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/jax/presubmit.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "jax"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/keras2/continuous.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "1"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/keras2/presubmit.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "1"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/tensorflow/continuous.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "tensorflow"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/tensorflow/presubmit.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "tensorflow"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/torch/continuous.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "torch"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
18 changes: 18 additions & 0 deletions .kokoro/github/ubuntu/gpu/torch/presubmit.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,23 @@ env_vars: {
value: "torch"
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_username"
}
}
}

before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73361
keyname: "keras_kaggle_secret_key"
}
}
}

# Set timeout to 60 mins from default 180 mins
timeout_mins: 60
3 changes: 2 additions & 1 deletion keras_nlp/models/gemma/gemma_backbone_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,9 @@ def test_saved_model(self):
)

@pytest.mark.kaggle_key_required
@pytest.mark.large
@pytest.mark.extra_large
def test_smallest_preset(self):
# TODO: Fails with OOM on current GPU CI
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option would be to take in a dtype to run_preset_test and pass is it as kwarg to from_preset. Then we could try bfloat16 precision, which might avoid the oom?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried keras.config.set_floatx("float16"), but that didn't help. So going to land this PR now. We can address it in another PR

self.run_preset_test(
cls=GemmaBackbone,
preset="gemma_2b_en",
Expand Down
Loading