Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report stale metadata asset #28903

Merged
merged 10 commits into from
Aug 4, 2023
Merged

Report stale metadata asset #28903

merged 10 commits into from
Aug 4, 2023

Conversation

bnchrch
Copy link
Contributor

@bnchrch bnchrch commented Aug 1, 2023

Overview

Screenshot 2023-08-01 at 12 10 17 AM

Adds a new report to dagster that shows if files in the github repo are out of date with latest

This is part of #27853

is waiting on code in #28759 to complete the slack notification portion

@bnchrch bnchrch changed the title Bnchrch/stale metadata asset Report stale metadata asset Aug 1, 2023
@bnchrch bnchrch requested review from alafanechere and a team August 1, 2023 15:37
Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great straightforward implementation. Just wondiering how the interaction with github files works.



@asset(required_resource_keys={"latest_metadata_file_blobs"}, group_name=GROUP_NAME)
def stale_github_metadata_file(context, github_metadata_file_md5s: dict) -> OutputDataFrame:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neat function 👏

@bnchrch bnchrch requested review from a team as code owners August 3, 2023 01:14
@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit labels Aug 3, 2023
@octavia-squidington-iii
Copy link
Collaborator

destination-databend test report (commit 5c643c9dac) - ❌

⏲️ Total pipeline duration: 02mn07s

Step Result
Validate airbyte-integrations/connectors/destination-databend/metadata.yaml
Connector version semver check
QA checks
Code format checks
Connector package install
Build destination-databend docker image for platform linux/x86_64
Unit tests
Integration tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-databend test

@octavia-squidington-iii
Copy link
Collaborator

destination-convex test report (commit 5c643c9dac) - ✅

⏲️ Total pipeline duration: 02mn00s

Step Result
Validate airbyte-integrations/connectors/destination-convex/metadata.yaml
Connector version semver check
QA checks
Code format checks
Connector package install
Build destination-convex docker image for platform linux/x86_64
Unit tests
Integration tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-convex test

@octavia-squidington-iii
Copy link
Collaborator

destination-amazon-sqs test report (commit 5c643c9dac) - ✅

⏲️ Total pipeline duration: 02mn21s

Step Result
Validate airbyte-integrations/connectors/destination-amazon-sqs/metadata.yaml
Connector version semver check
QA checks
Code format checks
Connector package install
Build destination-amazon-sqs docker image for platform linux/x86_64
Unit tests
Integration tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-amazon-sqs test

@octavia-squidington-iii
Copy link
Collaborator

destination-aws-datalake test report (commit 5c643c9dac) - ✅

⏲️ Total pipeline duration: 03mn41s

Step Result
Validate airbyte-integrations/connectors/destination-aws-datalake/metadata.yaml
Connector version semver check
QA checks
Code format checks
Connector package install
Build destination-aws-datalake docker image for platform linux/x86_64
Unit tests
Integration tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-aws-datalake test

@bnchrch bnchrch force-pushed the bnchrch/stale-metadata-asset branch from 5c643c9 to 40ae367 Compare August 3, 2023 01:34
@octavia-squidington-iii octavia-squidington-iii removed CDK Connector Development Kit area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Aug 3, 2023
@bnchrch bnchrch enabled auto-merge (squash) August 4, 2023 20:27
@bnchrch bnchrch merged commit f07dffa into master Aug 4, 2023
@bnchrch bnchrch deleted the bnchrch/stale-metadata-asset branch August 4, 2023 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants