Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto Tagger Application - Preparing the Ingestion Framework #13862

Merged
merged 17 commits into from
Nov 13, 2023

Conversation

pmbrull
Copy link
Collaborator

@pmbrull pmbrull commented Nov 6, 2023

Describe your changes:

This is the first iteration for a more generic approach of External Apps:

  1. We have an ExternalApplicationHandler in the backend that should be able to work with any external app (Data Insights will stay as-is)
  2. The ExternalApplicationHandler will create an Ingestion Pipeline of type Application that translates to an ApplicationWorkflow, as a simplified ingestion workflow with a single step
  3. Added a metadata app -c <path> CLI tool to play around with apps from YAML
  4. Did the first iteration of the AutoTagger where we are just translating the PII processor from the profiler. We will iterate on this to handle the generic tagging feature

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@github-actions github-actions bot added UI UI specific issues Ingestion backend safe to test Add this label to run secure Github workflows on PRs labels Nov 6, 2023
Copy link
Contributor

github-actions bot commented Nov 6, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link
Contributor

github-actions bot commented Nov 6, 2023

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 48%
48.8% (23646/48457) 30.43% (8851/29089) 30.26% (2679/8853)

Copy link
Contributor

github-actions bot commented Nov 7, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link
Contributor

github-actions bot commented Nov 8, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@pmbrull pmbrull changed the title Auto PII Application - Preparing the Ingestion Framework Auto Tagger Application - Preparing the Ingestion Framework Nov 8, 2023
@pmbrull pmbrull marked this pull request as ready for review November 8, 2023 14:38
@harshach harshach self-assigned this Nov 8, 2023
@pmbrull pmbrull requested a review from a team November 9, 2023 08:56
Copy link

[open-metadata-ui] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

Copy link

[open-metadata-airflow-apis] SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

Copy link

[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

49.5% 49.5% Coverage
4.2% 4.2% Duplication

@pmbrull pmbrull merged commit c742835 into open-metadata:main Nov 13, 2023
19 of 21 checks passed
@pmbrull pmbrull deleted the waii-app branch November 13, 2023 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Ingestion safe to test Add this label to run secure Github workflows on PRs UI UI specific issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants