-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MINOR - Clean up configs & add auto classification docs #18907
Conversation
@@ -1745,7 +1745,7 @@ WHERE JSON_EXTRACT(json, '$.pipelineType') = 'metadata'; | |||
|
|||
-- classification and sampling configs from the profiler pipelines | |||
UPDATE ingestion_pipeline_entity | |||
SET json = JSON_REMOVE(json, '$.sourceConfig.config.processPiiSensitive', '$.sourceConfig.config.confidence', '$.sourceConfig.config.generateSampleData') | |||
SET json = JSON_REMOVE(json, '$.sourceConfig.config.processPiiSensitive', '$.sourceConfig.config.confidence', '$.sourceConfig.config.generateSampleData', '$.sourceConfig.config.sampleDataCount') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is also not needed for the profiler. It was only used for sample data
session = self.create_session() | ||
if self.config.endPointURL is not None: | ||
return session.client( | ||
service_name=service_name, endpoint_url=str(self.config.endPointURL) | ||
) | ||
return session.client(service_name=service_name) | ||
|
||
logger.info(f"Getting AWS default client for service [{service_name}]") | ||
logger.debug(f"Getting AWS default client for service [{service_name}]") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since we're now passing the client to the source externally, this log was too chatty
profile_sample_type=self.source_config.profileSampleType, | ||
sampling_method_type=self.source_config.samplingMethodType, | ||
), | ||
default_sample_config=SampleConfig(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keeping simpler pipeline for the auto classification. If there's anything configured for the table we'll pick it from there directly
Quality Gate passed for 'open-metadata-ui'Issues Measures |
Quality Gate passed for 'open-metadata-ingestion'Issues Measures |
Describe your changes:
Fixes
I worked on ... because ...
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>