feat(ingest): lineage for SageMaker model endpoints and groups #2894
Conversation
| "Deleting": EndpointStatusClass.DELETING, | ||
| "Failed": EndpointStatusClass.FAILED, | ||
| "Unknown": EndpointStatusClass.UNKNOWN, | ||
| } |
There was a problem hiding this comment.
do we need this level of granularity in our model as well?
There was a problem hiding this comment.
I think so – it may be a little too detailed for our current pull-based integration, but it will be useful once we have push-based functionality.
| @@ -55,6 +55,7 @@ def check_golden_file( | |||
| # if updating a golden file that doesn't exist yet, load the output again | |||
| if update_golden and not golden_exists: | |||
| golden = load_json_file(output_path) | |||
| shutil.copyfile(str(output_path), str(golden_path)) | |||
There was a problem hiding this comment.
If the golden doesn't exist, doesn't it make sense to just set it to None or [] -> there is a diff -> we update the underlying file
This logic seems a bit too complex, but we can address this in a follow up
| Ownership, | ||
| Status, | ||
| Deprecation, | ||
| BrowsePaths |
There was a problem hiding this comment.
will ml model endpoints be browsable? probably not, right? I imagine these will just be linked via ml models
There was a problem hiding this comment.
No they won't – will remove
| /** | ||
| * Name of the MLModelEndpoint | ||
| */ | ||
| @Searchable = { | ||
| "fieldType": "TEXT_PARTIAL", | ||
| "enableAutocomplete": true, | ||
| "boostScore": 10.0 | ||
| } | ||
| name: string |
There was a problem hiding this comment.
where do these names come from?
There was a problem hiding this comment.
SageMaker – you can set a name for an endpoint
metadata-models/src/main/pegasus/com/linkedin/ml/metadata/MLModelEndpointProperties.pdl
Outdated
Show resolved
Hide resolved
metadata-models/src/main/pegasus/com/linkedin/ml/metadata/MLModelEndpointProperties.pdl
Outdated
Show resolved
Hide resolved
metadata-models/src/main/pegasus/com/linkedin/ml/metadata/MLModelGroupProperties.pdl
Outdated
Show resolved
Hide resolved
metadata-models/src/main/pegasus/com/linkedin/ml/metadata/MLModelGroupProperties.pdl
Outdated
Show resolved
Hide resolved
Uses the SageMaker lineage graph to extract and infer lineage between models and endpoints as well as models and model groups.
Checklist