Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GEN-309: OpenAPI Connector #17754

Merged
merged 14 commits into from
Sep 11, 2024
Merged

GEN-309: OpenAPI Connector #17754

merged 14 commits into from
Sep 11, 2024

Conversation

harshsoni2024
Copy link
Contributor

@harshsoni2024 harshsoni2024 commented Sep 6, 2024 •
edited
Loading

Describe your changes:

Fixes

I worked on OpenAPI Ingestion Connector

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@notion-workspace Notion Workspace
Copy link

OpenAPI Metadata Ingestion Connector

@github-actions github-actions bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Sep 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 6, 2024 •
edited
Loading

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 65%
 65.12% (37220/57154) 41.61% (14804/35576) 43.94% (4529/10308)

pmbrull
pmbrull requested changes Sep 6, 2024
View reviewed changes
ingestion/src/metadata/ingestion/source/api/rest/connection.py Outdated
"""

def custom_url_exec():
if client.status_code == 200:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this enough? Aren't we expecting any specific shape? I guess that if I put www.google.com i might get a 200 but it's not a real service that I can parse no? 🤔

ingestion/src/metadata/ingestion/source/api/rest/metadata.py Outdated
"""Method to return api collection Entities"""
try:
collection_request = CreateAPICollectionRequest(
name=EntityName(collection.get("name")),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's prepare pydantic models, this is error-prone:

  1. collection.get("name") might be None
  2. EntityName does not allow None
  3. The error better tell us the data is not behaving properly, rather than enigmatic pydantic errors.

Let's properly parse the payload and validate early (shift left as many errors as possible), then let the rest of the flow be "dumb" on how to treat the results, instead of having to be aware all the time about the shape of our data

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed for 'open-metadata-airflow-apis'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
52.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@harshsoni2024 harshsoni2024 merged commit 4f57f4e into open-metadata:main Sep 11, 2024
37 of 38 checks passed
@harshsoni2024 harshsoni2024 deleted the GEN-309 branch December 19, 2024 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Sachin-chaurasiya Sachin-chaurasiya Sachin-chaurasiya approved these changes

@pmbrull pmbrull pmbrull approved these changes

@ShaileshParmar11 ShaileshParmar11 Awaiting requested review from ShaileshParmar11

@karanh37 karanh37 Awaiting requested review from karanh37

@chirag-madlani chirag-madlani Awaiting requested review from chirag-madlani

Assignees
No one assigned
Labels
backend Ingestion safe to test Add this label to run secure Github workflows on PRs spec UI UI specific issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@harshsoni2024 @pmbrull @Sachin-chaurasiya