ADS changes for MS enhancements #1036

vikasray0208 · 2025-01-09T09:21:44Z

Description

This PR is intended to add/update the following changes in ADS in accordance with AQUA MS enhancements

Listing service and custom models
CRUD APIs for defined and custom metadata artifact

This PR will enable following changes

Service Model Listing: Enable the listing of service models without requiring the service compartment ID and cross-tenancy access.
Metadata Storage: Enable additional metadata store , such as license content, README.md, deployment config and fine-tuning config. Also, implement a search functionality for the license and keys from the README via RQS.

Example


ds_model=DataScienceModel.from_id("ocid1.datasciencemodel.oc1.iad.xxxxx")

# creating defined metadata artifact for key DeploymentConfiguration
ds_model.create_defined_metadata_artifact("DeploymentConfiguration",deployment_config_path,"local")

# Downloading the defined metadata artifact
ds_model.get_defined_metadata_artifact("DeploymentConfiguration",target_dir="/Users/path/to/local/folder")

Unit tests with latest preview oci sdk

Version: 2.142.1+preview.1.1

References

UPDATE:

public oci sdk has been released: https://github.com/oracle/oci-python-sdk/releases/tag/v2.148.0

github-actions · 2025-01-09T12:02:51Z

📌 Cov diff with main:

📌 Overall coverage:

VipulMascarenhas

left some minor comments, overall looks good.

ads/model/datascience_model.py

ads/model/service/oci_datascience_model.py

…feature/aqua_ms_changes_2

ads/model/service/oci_datascience_model.py

The merge-base changed after approval.

github-actions · 2025-01-10T21:10:20Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-03-11T09:54:57Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-03-11T12:04:33Z

📌 Cov diff with main:

📌 Overall coverage:

mrDzurb · 2025-03-11T18:27:38Z

ads/common/utils.py

@@ -82,6 +85,14 @@
    color=["teal", "blueviolet", "forestgreen", "peru", "y", "dodgerblue", "r"]
 )

+
+# Metadata artifact path type can be either local path or OSS path. It can also be the content itself.
+class MetadataArtifactPathType(ExtendedEnum):


If the class will be used only within ads.models, then it might make sense to move it to the ads.models.common.utils.py?

Updated. Thanks!

mrDzurb · 2025-03-11T19:21:05Z

ads/common/utils.py

@@ -212,6 +239,15 @@ def random_valid_ocid(prefix="ocid1.dataflowapplication.oc1.iad"):
    return f"{left}.{fake}"


+def read_file(file_path: str, **kwargs) -> str:


Could you double check, we might already have something similar in common utils

I didn't create this function , just moved it from ads.aqua.common.utils to ads.common.utils due to circular import issue.
Link to the comment: #1036 (comment)

mrDzurb · 2025-03-11T19:39:56Z

ads/model/model_metadata.py

        if isinstance(key_value_map["value"], str):
            try:
                key_value_map["value"] = json.loads(oci_metadata_item.get("value"))
+                key_value_map["has_artifact"] = bool(


I think we should use some bool parser method for this?

def parse_bool(value: Any) -> bool: """ Converts a value to boolean. For strings, it interprets 'true', '1', or 'yes' (case insensitive) as True; everything else as False. Parameters ---------- value : Any The value to convert to boolean. Returns ------- bool The boolean interpretation of the value. """ if isinstance(value, bool): return value if isinstance(value, str): return value.strip().lower() in ("true", "1", "yes") return bool(value)

Updated. Thanks!

github-actions · 2025-03-11T21:55:08Z

📌 Cov diff with main:

📌 Overall coverage:

lu-ohai · 2025-03-11T22:07:34Z

ads/model/datascience_model.py

+
+        Returns
+        -------
+        Dict


Some of the comments in this file are still incorrect, for instance, the return type of this API is ModelMetadataArtifactDetails, notDict. Could you update all of them?

Updated. Thanks!

lu-ohai · 2025-03-11T22:09:38Z

ads/model/datascience_model.py

+        )
+
+    def get_custom_metadata_artifact(
+        self, metadata_key_name: str, target_dir: str, override: bool = False


Would it be better if we have some logic in this API to return the file_content directly if no target_dir is provided? @mrDzurb @mayoor

I like this idea, we can do this in both cases, even if the target_dir provided, we can still return the content.

Updated. Thanks!

ads/model/service/oci_datascience_model.py

ads/model/datascience_model.py

github-actions · 2025-03-11T23:00:43Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-03-12T08:47:06Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-03-12T15:03:56Z

📌 Cov diff with main:

📌 Overall coverage:

lu-ohai · 2025-03-12T19:22:01Z

ads/aqua/constants.py

@@ -3,7 +3,6 @@
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 """This module defines constants used in ads.aqua module."""

-UNKNOWN = ""


Why do we remove UNKNOWN here, it causes some error in feature/multi_model_deploy branch. I'll add it back to the corresponding pr. @mrDzurb

It was moved to ads.common.utils since this function was moved to ads.common.utils to avoid circular imports

def read_file(file_path: str, **kwargs) -> str: try: with fsspec.open(file_path, "r", **kwargs.get("auth", {})) as f: return f.read() except Exception as e: logger.debug(f"Failed to read file {file_path}. {e}") return UNKNOWN

Since this is a generic function , it was adviced to move this to ads.common.utils

vikasray0208 added 2 commits January 9, 2025 14:46

Added methods for aqua-model store apis change

d681e17

update modela and mvs list api

77783ce

vikasray0208 requested review from darenr, mayoor, mrDzurb, VipulMascarenhas, qiuosier and ahosler as code owners January 9, 2025 09:21

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jan 9, 2025

vikasray0208 added 3 commits January 9, 2025 14:54

update the test file

804d17f

addressed review comments

09f2f9b

fix unit test

3dd7b85

VipulMascarenhas reviewed Jan 10, 2025

View reviewed changes

vikasray0208 added 10 commits January 10, 2025 13:00

addressed review comments

0aa0df0

fix unit test erro

663e7b2

Added methods for aqua-model store apis change

8f7da9e

update modela and mvs list api

9207cce

update the test file

32d5695

addressed review comments

446df19

fix unit test

0ea9c9d

addressed review comments

44e04bd

fix unit test erro

bae144e

Merge remote-tracking branch 'origin/feature/aqua_ms_changes_2' into …

db59823

…feature/aqua_ms_changes_2

VipulMascarenhas previously approved these changes Jan 10, 2025

View reviewed changes

ads/model/service/oci_datascience_model.py Outdated Show resolved Hide resolved

Merge branch 'main' into feature/aqua_ms_changes_2

ca4b8c9

mrDzurb previously approved these changes Jan 11, 2025

View reviewed changes

VipulMascarenhas previously approved these changes Jan 11, 2025

View reviewed changes

Merge branch 'main' into feature/aqua_ms_changes_2

c1b1b96

kumar-shivam-ranjan self-assigned this Mar 11, 2025

kumar-shivam-ranjan requested review from mayoor and lu-ohai March 11, 2025 07:32

Merge branch 'main' into feature/aqua_ms_changes_2

1b87417

Reverting unwanted changes pushed by mistake

40456fe

kumar-shivam-ranjan self-requested a review March 11, 2025 16:54

mrDzurb reviewed Mar 11, 2025

View reviewed changes

kumar-shivam-ranjan added 4 commits March 12, 2025 02:21

Addressing review comments

45e9014

Fixing UTs

782839f

Adding bool parser

aabbb98

Fixing UTs

b4999d0

Merge branch 'main' into feature/aqua_ms_changes_2

192248a

lu-ohai reviewed Mar 11, 2025

View reviewed changes

Addressing review comments

54296f1

lu-ohai reviewed Mar 11, 2025

View reviewed changes

ads/model/datascience_model.py Show resolved Hide resolved

kumar-shivam-ranjan added 2 commits March 12, 2025 03:51

Addressing review comments

39763e9

Updating BytesIO to bytes

2eb12a3

Adding path directory validation

e414716

Merge branch 'main' into feature/aqua_ms_changes_2

5ef6f12

mrDzurb approved these changes Mar 12, 2025

View reviewed changes

mayoor approved these changes Mar 12, 2025

View reviewed changes

kumar-shivam-ranjan merged commit e7e3f19 into main Mar 12, 2025
20 of 21 checks passed

lu-ohai reviewed Mar 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADS changes for MS enhancements #1036

ADS changes for MS enhancements #1036

vikasray0208 commented Jan 9, 2025 •

edited by kumar-shivam-ranjan

Loading

github-actions bot commented Jan 9, 2025

VipulMascarenhas left a comment

github-actions bot commented Jan 10, 2025

github-actions bot commented Mar 11, 2025

github-actions bot commented Mar 11, 2025

mrDzurb Mar 11, 2025

kumar-shivam-ranjan Mar 11, 2025

kumar-shivam-ranjan Mar 11, 2025

mrDzurb Mar 11, 2025

kumar-shivam-ranjan Mar 11, 2025

mrDzurb Mar 11, 2025

kumar-shivam-ranjan Mar 11, 2025

github-actions bot commented Mar 11, 2025

lu-ohai Mar 11, 2025

kumar-shivam-ranjan Mar 11, 2025

lu-ohai Mar 11, 2025

mrDzurb Mar 11, 2025

kumar-shivam-ranjan Mar 12, 2025

github-actions bot commented Mar 11, 2025

github-actions bot commented Mar 12, 2025

github-actions bot commented Mar 12, 2025

lu-ohai Mar 12, 2025 •

edited

Loading

kumar-shivam-ranjan Mar 13, 2025 •

edited

Loading

kumar-shivam-ranjan Mar 13, 2025

		@@ -212,6 +239,15 @@ def random_valid_ocid(prefix="ocid1.dataflowapplication.oc1.iad"):
		return f"{left}.{fake}"


		def read_file(file_path: str, **kwargs) -> str:

ADS changes for MS enhancements #1036

ADS changes for MS enhancements #1036

Conversation

vikasray0208 commented Jan 9, 2025 • edited by kumar-shivam-ranjan Loading

Description

Example

Unit tests with latest preview oci sdk

References

UPDATE:

github-actions bot commented Jan 9, 2025

VipulMascarenhas left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2025

github-actions bot commented Mar 11, 2025

github-actions bot commented Mar 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 11, 2025

github-actions bot commented Mar 12, 2025

github-actions bot commented Mar 12, 2025

lu-ohai Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

kumar-shivam-ranjan Mar 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vikasray0208 commented Jan 9, 2025 •

edited by kumar-shivam-ranjan

Loading

lu-ohai Mar 12, 2025 •

edited

Loading

kumar-shivam-ranjan Mar 13, 2025 •

edited

Loading