Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for external profiler workflow #13887

Merged
merged 6 commits into from
Nov 9, 2023
Merged

Add support for external profiler workflow #13887

merged 6 commits into from
Nov 9, 2023

Conversation

ulixius9
Copy link
Member

@ulixius9 ulixius9 commented Nov 7, 2023

Describe your changes:

Fixes

I worked on ... because ...

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@ulixius9 ulixius9 requested a review from a team as a code owner November 7, 2023 16:25
@github-actions github-actions bot added Ingestion backend safe to test Add this label to run secure Github workflows on PRs labels Nov 7, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 7, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@ulixius9 ulixius9 force-pushed the profiler_ext_workflow branch from 8cffc4d to 2c5e527 Compare November 8, 2023 06:20
pmbrull
pmbrull reviewed Nov 8, 2023
View reviewed changes
ingestion/src/metadata/profiler/source/metadata_ext.py Outdated Show resolved Hide resolved
ingestion/src/metadata/workflow/profiler.py Show resolved Hide resolved
ingestion/src/metadata/workflow/usage.py Outdated
@@ -44,6 +45,10 @@ def set_steps(self):
def _get_source(self) -> Source:
# Source that we are ingesting, e.g., mysql, looker or kafka
source_type = self.config.source.type.lower()
if not self.config.source.serviceName:
raise WorkflowExecutionError(
"ServiceName is required field for executing the Metadata Workflow"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be needed in the lineage workflow as well correct?

We could word it as

`serviceName` is a required field for executing the workflow. 
You can find more information on how to build the YAML configuration here: <link to docs for each workflow>

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not solved yet right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh there is no "lineage" workflow right we use metadata workflow for lineage ain't we?

ingestion/src/metadata/ingestion/source/database/azuresql/metadata.py Show resolved Hide resolved
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 8, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Nov 8, 2023

[open-metadata-ingestion] SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

26.1% 26.1% Coverage
10.9% 10.9% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@ulixius9 ulixius9 requested a review from pmbrull November 8, 2023 13:16
@ulixius9 ulixius9 merged commit 4c19bb5 into main Nov 9, 2023
8 of 11 checks passed
@ulixius9 ulixius9 deleted the profiler_ext_workflow branch November 9, 2023 13:20
ulixius9 added a commit that referenced this pull request Nov 10, 2023
@ulixius9
Add support for external profiler workflow (#13887)
153084d 
* Add support for external profiler workflow

* pylint

* resolve comments

* resolve comments

* pyformat

* fix code smell
ulixius9 added a commit that referenced this pull request Nov 10, 2023
@ulixius9
Add support for external profiler workflow (#13887)
f87649d 
* Add support for external profiler workflow

* pylint

* resolve comments

* resolve comments

* pyformat

* fix code smell
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pmbrull pmbrull pmbrull approved these changes

Assignees
No one assigned
Labels
backend Ingestion safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ulixius9 @pmbrull