Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Support complex metrics in Vertex Experiments #1698

Merged
merged 21 commits into from
Sep 30, 2022
Merged

feat: Support complex metrics in Vertex Experiments #1698

merged 21 commits into from
Sep 30, 2022

Conversation

jaycee-li
Copy link
Contributor

Vertex Experiments OKR:
[P0] Launch support for visualization of confusion matrices
[P0] Launch support for visualization of AUC/ROC curves

Design Doc: https://docs.google.com/document/d/1LpO7hl2z6d_8smvhNXmGDXrS89A-j8g7ew6j5bkXmQE/edit?resourcekey=0-p42-rx_ZNnHS-WxKHVxkFQ#heading=h.foacqi32i926

jaycee-li and others added 2 commits September 26, 2022 09:00
@jaycee-li
Experiments complex metrics (#8)
3ca2d28 
* feat: new class and API for metrics

* update system test

* update high level log method

* fix system test

* update example

* change from system schema to google schema
@jaycee-li
fix: import error
ee36af9
@jaycee-li jaycee-li requested review from a team as code owners September 26, 2022 16:24
@product-auto-label product-auto-label bot added the size: l Pull request size is large. label Sep 26, 2022
@snippet-bot
Copy link

snippet-bot bot commented Sep 26, 2022 •
edited
Loading

Here is the summary of changes.

You are about to add 2 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@product-auto-label product-auto-label bot added the api: vertex-ai Issues related to the googleapis/python-aiplatform API. label Sep 26, 2022
dandhlee
dandhlee approved these changes Sep 26, 2022
View reviewed changes
Copy link
Contributor

@dandhlee dandhlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for Samples.

samples/model-builder/experiment_tracking/log_classification_metrics_sample.py Show resolved Hide resolved
samples/model-builder/experiment_tracking/log_classification_metrics_sample.py Outdated Show resolved Hide resolved
@jaycee-li jaycee-li added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 26, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 26, 2022
dandhlee
dandhlee reviewed Sep 26, 2022
View reviewed changes
Copy link
Contributor

@dandhlee dandhlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Sounds good.

samples/model-builder/conftest.py Show resolved Hide resolved
@jaycee-li @dandhlee
Update samples/model-builder/conftest.py
5bf0d13 
Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com>
@sasha-gitg sasha-gitg self-requested a review September 26, 2022 19:29
@sasha-gitg sasha-gitg self-assigned this Sep 26, 2022
@jaycee-li jaycee-li added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 27, 2022
sasha-gitg
sasha-gitg requested changes Sep 28, 2022
View reviewed changes
Copy link
Member

@sasha-gitg sasha-gitg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work Jaycee! Mainly LGTM. Minor comments. Thanks!

google/cloud/aiplatform/metadata/experiment_run_resource.py Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/metadata/experiment_run_resource.py Outdated Show resolved Hide resolved
tests/unit/aiplatform/test_metadata.py Show resolved Hide resolved
@jaycee-li
Copy link
Contributor Author

@sasha-gitg all the comments have been addressed. Please review again. Thank you!

@jaycee-li
Copy link
Contributor Author

Hi @SinaChavoshi @KevinBNaughton , I've updated the ClassificationMetrics class. Please review google/cloud/aiplatform/metadata/schema/google/artifact_schema.py, google/cloud/aiplatform/metadata/schema/utils.py, and tests/unit/aiplatform/test_metadata_schema.py. Thank you!

KevinBNaughton
KevinBNaughton approved these changes Sep 29, 2022
View reviewed changes
Copy link
Contributor

@KevinBNaughton KevinBNaughton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked through the schema/google and schema/utils.py and they look good to me. Thanks for adding this! Can we wait to submit this PR until the regions are backfilled, or tested on autopush first to make sure that backfilling regions does not cause any issues?

@jaycee-li
Copy link
Contributor Author

I looked through the schema/google and schema/utils.py and they look good to me. Thanks for adding this! Can we wait to submit this PR until the regions are backfilled, or tested on autopush first to make sure that backfilling regions does not cause any issues?

Thanks for the review! We plan to submit this PR today or tomorrow, and launch this feature next Monday. Otherwise the release freeze starts and we won't be able to launch it until Nov. Do you know when will we backfill regions?

SinaChavoshi
SinaChavoshi approved these changes Sep 29, 2022
View reviewed changes
Copy link
Contributor

@SinaChavoshi SinaChavoshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! few minor comments otherwise looks good to me.

google/cloud/aiplatform/metadata/schema/utils.py Show resolved Hide resolved
google/cloud/aiplatform/metadata/schema/google/artifact_schema.py Show resolved Hide resolved
tests/unit/aiplatform/test_metadata_schema.py Show resolved Hide resolved
@jaycee-li jaycee-li merged commit ed0492e into googleapis:main Sep 30, 2022
@jaycee-li jaycee-li deleted the complex-metrics branch September 30, 2022 20:13
@release-please release-please bot mentioned this pull request Sep 30, 2022
Merged
@release-please release-please bot mentioned this pull request Jun 8, 2023
Closed
@release-please release-please bot mentioned this pull request Aug 25, 2023
Closed
@release-please release-please bot mentioned this pull request Sep 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dandhlee dandhlee dandhlee approved these changes

@sasha-gitg sasha-gitg sasha-gitg approved these changes

@KevinBNaughton KevinBNaughton KevinBNaughton approved these changes

@SinaChavoshi SinaChavoshi SinaChavoshi approved these changes

@nicain nicain Awaiting requested review from nicain nicain was automatically assigned from googleapis/python-samples-reviewers

Assignees

@sasha-gitg sasha-gitg

Labels
api: vertex-ai Issues related to the googleapis/python-aiplatform API. size: l Pull request size is large.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@jaycee-li @SinaChavoshi @KevinBNaughton @sasha-gitg @dandhlee @yoshi-kokoro