Source Github: add workflow_jobs stream #16534

Mainara · 2022-09-09T13:39:58Z

What

This PR adds a new stream for github source: workflow_jobs

How

Used the endpoint: https://docs.github.com/pt/rest/actions/workflow-jobs#list-jobs-for-a-workflow-run to lists jobs for a workflow run and create a stream.

User Impact

There is no breakchanges, just a new stream was added.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Tests

Unit

airbyte-integrations/connectors/source-github/unit_tests/test_stream.py::test_stream_workflow_jobs_read_incremental ✓
airbyte-integrations/connectors/source-github/unit_tests/test_stream.py::test_stream_workflow_jobs_full_refresh ✓

…s from Github;

…t to method incremetal in unit test to class 'WorkflowJobs';

…'WorkflowJobs';

CLAassistant · 2022-09-09T13:40:26Z

All committers have signed the CLA.

sajarin · 2022-09-09T16:00:19Z

Hi, thanks for the PR. This PR will be reviewed by one of our community maintainers as part our of Community Maintainer Program. Someone will be assigned shortly, thanks for being patient!

Mainara · 2022-09-09T17:53:26Z

This PR is related to this issue: #16360

YiyangLi · 2022-09-14T10:38:00Z

/test connector=connectors/github

sajarin · 2022-09-14T17:21:40Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3054767833
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3054767833
🐛 https://gradle.com/s/ypjdx2era5q4s

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
=================== 1 failed, 38 passed in 292.94s (0:04:52) ===================

YiyangLi · 2022-09-14T23:47:48Z

@Mainara Thanks for your contribution. It looks like the Workflow_jobs stream supports incremental sync, which will run the test called TestStateWithAbnormallyLargeValues, here is the documentation.

Basically, it will read the file abnormal state from the json file, and it expects to return zero results because the state value is abnormally huge. Can you add a state value for workflow_jobs?

…and unit test;

YiyangLi · 2022-09-17T14:47:56Z

/test connector=connectors/source-github

…n' for integration tests;

py-paulo · 2022-09-19T14:10:07Z

@YiyangLi we made the required changes, can you please validate it?

sajarin · 2022-09-19T14:30:16Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3083224659
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3083224659
Python tests coverage:

Name                             Stmts   Miss  Cover
----------------------------------------------------
source_github/utils.py              14      0   100%
source_github/github_schema.py    8807      0   100%
source_github/__init__.py            2      0   100%
source_github/graphql.py           145      1    99%
source_github/streams.py           779     39    95%
source_github/source.py            104     27    74%
----------------------------------------------------
TOTAL                             9851     67    99%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1325    463    65%

Build Passed

Test summary info:

All Passed

sajarin · 2022-09-19T18:08:18Z

/publish connector=connectors/source-github

🕑 Publishing the following connectors:
connectors/source-github
https://github.com/airbytehq/airbyte/actions/runs/3084608070

Connector	Did it publish?	Were definitions generated?
connectors/source-github	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Mainara · 2022-09-19T21:14:49Z

/publish connector=connectors/source-github auto-bump-version=false

Mainara · 2022-09-20T16:37:21Z

@sajarin We verify that the image of the new version has already been published: airbyte/source-github:0.3.0, and the error is precisely because he tried to publish again this version. What can we do with this situation?

davydov-d · 2022-09-21T09:37:42Z

hey @sajarin can we please merge this PR? looks like it's completed.
it currently blocks from testing new ones(https://github.com/airbytehq/airbyte/actions/runs/3096389815/jobs/5011926155) as the image is published but not merged

sajarin · 2022-09-21T14:20:59Z

Should be good to go @davydov-d.

Thanks for the PR @Mainara and @py-paulo and thanks for the review @YiyangLi!

* feat(workflow-jobs)!: new class 'WorkflowJobs' to get job informations from Github; * feat(workflow-jobs)!: unit test to new class 'WorkflowJobs'; * feat(test_stream/test_stream_workflow_jobs_read_incremental)!: support to method incremetal in unit test to class 'WorkflowJobs'; * feat(test_workflow): unit test incremental and full refresh to class 'WorkflowJobs'; * docs: updated github docs * docs: updated github docs * docs: updated github.md * docs: updated github.md * fix: add workflow_jobs in abnormal tests, use 'start date' in source and unit test; * fix: add workflow_jobs in 'abnormal_state.json' and 'sample_state.json' for integration tests; Co-authored-by: Paulo Roberto <paulo.rb.beserra@gmail.com> Co-authored-by: Paulo Roberto <71179739+py-paulo@users.noreply.github.com>

py-paulo and others added 6 commits September 7, 2022 19:54

feat(workflow-jobs)!: new class 'WorkflowJobs' to get job information…

b905daa

…s from Github;

feat(workflow-jobs)!: unit test to new class 'WorkflowJobs';

f815643

feat(test_stream/test_stream_workflow_jobs_read_incremental)!: suppor…

4106794

…t to method incremetal in unit test to class 'WorkflowJobs';

feat(test_workflow): unit test incremental and full refresh to class …

96ea919

…'WorkflowJobs';

docs: updated github docs

400912d

docs: updated github docs

715d5fb

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 9, 2022

octavia-squidington-iii added bounty community connectors/source/github labels Sep 9, 2022

Mainara added 3 commits September 9, 2022 10:53

docs: updated github.md

e9c0272

docs: updated github.md

2b7bfcf

Merge branch 'master' into github-workflow-jobs

277e727

sajarin changed the title ~~Github source: add workflow_jobs stream~~ Source Github: add workflow_jobs stream Sep 9, 2022

sajarin added the bounty-L Maintainer program: claimable large bounty PR label Sep 9, 2022

sajarin mentioned this pull request Sep 13, 2022

Source Github: Support workflow jobs #16588

Closed

37 tasks

py-paulo and others added 3 commits September 16, 2022 16:03

Merge branch 'master' into github-workflow-jobs

bceca40

Merge branch 'master' into github-workflow-jobs

ed2f6a9

fix: add workflow_jobs in abnormal tests, use 'start date' in source …

8e7bc2a

…and unit test;

py-paulo and others added 3 commits September 17, 2022 20:32

Merge branch 'master' into github-workflow-jobs

ca40604

fix: add workflow_jobs in 'abnormal_state.json' and 'sample_state.jso…

2898c2e

…n' for integration tests;

Merge branch 'master' into github-workflow-jobs

7a7c552

Merge branch 'master' into github-workflow-jobs

9d111c7

sajarin assigned YiyangLi Sep 19, 2022

sajarin approved these changes Sep 19, 2022

View reviewed changes

sajarin unassigned YiyangLi Sep 19, 2022

Merge branch 'master' into github-workflow-jobs

f96bebf

py-paulo added 2 commits September 19, 2022 23:19

Merge branch 'master' into github-workflow-jobs

23c3db9

Merge branch 'master' into github-workflow-jobs

581e551

davydov-d mentioned this pull request Sep 21, 2022

Source github: improve error logging #16947

Merged

sajarin merged commit 97b9c74 into airbytehq:master Sep 21, 2022

octavia-squidington-iii mentioned this pull request Sep 21, 2022

Bump Airbyte version from 0.40.7 to 0.40.8 #17009

Merged

sajarin assigned YiyangLi Sep 23, 2022

bleonard added the reward-150 label Oct 31, 2022

lazebnyi mentioned this pull request May 4, 2023

Source Github: get data from workflow jobs #16360

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Github: add workflow_jobs stream #16534

Source Github: add workflow_jobs stream #16534

Mainara commented Sep 9, 2022 •

edited

Loading

CLAassistant commented Sep 9, 2022 •

edited

Loading

sajarin commented Sep 9, 2022

Mainara commented Sep 9, 2022

YiyangLi commented Sep 14, 2022

sajarin commented Sep 14, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Sep 14, 2022

YiyangLi commented Sep 17, 2022

py-paulo commented Sep 19, 2022

sajarin commented Sep 19, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 19, 2022 •

edited by github-actions bot

Loading

Mainara commented Sep 19, 2022

Mainara commented Sep 20, 2022

davydov-d commented Sep 21, 2022

sajarin commented Sep 21, 2022

Source Github: add workflow_jobs stream #16534

Source Github: add workflow_jobs stream #16534

Conversation

Mainara commented Sep 9, 2022 • edited Loading

What

How

Recommended reading order

User Impact

Pre-merge Checklist

Community member or Airbyter

Tests

CLAassistant commented Sep 9, 2022 • edited Loading

sajarin commented Sep 9, 2022

Mainara commented Sep 9, 2022

YiyangLi commented Sep 14, 2022

sajarin commented Sep 14, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Sep 14, 2022

YiyangLi commented Sep 17, 2022

py-paulo commented Sep 19, 2022

sajarin commented Sep 19, 2022 • edited by github-actions bot Loading

Build Passed

sajarin commented Sep 19, 2022 • edited by github-actions bot Loading

Mainara commented Sep 19, 2022

Mainara commented Sep 20, 2022

davydov-d commented Sep 21, 2022

sajarin commented Sep 21, 2022

Mainara commented Sep 9, 2022 •

edited

Loading

CLAassistant commented Sep 9, 2022 •

edited

Loading

sajarin commented Sep 14, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 19, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 19, 2022 •

edited by github-actions bot

Loading