Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new Auto Classification Workflow #18610

Merged
merged 36 commits into from
Nov 19, 2024

Conversation

pmbrull
Copy link
Collaborator

@pmbrull pmbrull commented Nov 12, 2024

Describe your changes:

  • We are separating the sampler out of the Profiler. Now, the sampler is an input parameter for both Profiler and Test Suite. It will help us have the new sampling workflow
  • We are removing the factories for the sampler and test suite interface, and relying on the Service Spec instead
  • New Auto Classification Workflow and implementation for the IngestionPipeline
  • Removed processPIISensitive, confidence and generateSampleData from the profiler

I need to port the implementation for the ingestion to Argo in Collate https://github.com/open-metadata/openmetadata-collate/pull/923

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Copy link
Contributor

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link
Contributor

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link

Quality Gate Failed Quality Gate failed for 'open-metadata-airflow-apis'

Failed conditions
0.0% Coverage on New Code (required ≥ 20%)

See analysis details on SonarQube Cloud

Copy link

Quality Gate Failed Quality Gate failed for 'open-metadata-ingestion'

Failed conditions
E Security Review Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

@pmbrull pmbrull requested a review from TeddyCr November 19, 2024 05:57
@pmbrull pmbrull merged commit c68a45e into open-metadata:main Nov 19, 2024
17 of 28 checks passed
@pmbrull pmbrull deleted the separate-workflow branch November 19, 2024 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Backward-Incompatible-Change Ingestion safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants