🐛 Source Github: enable caching only for parent streams #13558

grubberr · 2022-06-07T15:08:12Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

python3 main.py read --config secrets/config.json --catalog catalog.json.workflow_runs > /dev/null

workflow_runs stream VCR caching Enabled

Resident memory peaks:

87132 kb
113976 kb
136844 kb
174036 kb
245368 kb
415032 kb
746760 kb
981160 kb
1.0g

workflow_runs stream VCR caching Disabled

Resident memory peaks:

49672 kb
49500 kb
....

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

codecov · 2022-06-07T15:20:41Z

Codecov Report

❗ No coverage uploaded for pull request base (master@bb46941). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #13558   +/-   ##
=========================================
  Coverage          ?   92.05%           
=========================================
  Files             ?        4           
  Lines             ?      680           
  Branches          ?        0           
=========================================
  Hits              ?      626           
  Misses            ?       54           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb46941...3d079db. Read the comment docs.

grubberr · 2022-06-07T15:42:03Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2455742905
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2455742905
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_github/utils.py          6      0   100%
source_github/__init__.py       2      0   100%
source_github/streams.py      572     28    95%
source_github/source.py        92     27    71%
-----------------------------------------------
TOTAL                         672     55    92%

Build Passed

Test summary info:

All Passed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-06-08T17:28:36Z

/test connector=connectors/source-github

grubberr · 2022-06-08T20:12:02Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2464098655
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2464098655
🐛 https://gradle.com/s/aa5kjxhwbvd4u

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
=================== 1 failed, 30 passed in 218.58s (0:03:38) ===================

grubberr · 2022-06-08T20:32:04Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2464204166
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2464204166
🐛 https://gradle.com/s/dibhufgkfvnxg

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
================== 1 failed, 30 passed in 1643.31s (0:27:23) ===================

grubberr · 2022-06-09T03:17:55Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2465713045
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2465713045
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_github/utils.py          6      0   100%
source_github/__init__.py       2      0   100%
source_github/streams.py      580     27    95%
source_github/source.py        92     27    71%
-----------------------------------------------
TOTAL                         680     54    92%

Build Passed

Test summary info:

All Passed

grubberr · 2022-06-09T03:44:19Z

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2465800600
🚀 Successfully published connectors/source-github
🚀 Auto-bumped version for connectors/source-github
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/2465800600

Cache only parent streams

d8bc2b0

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Jun 7, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jun 7, 2022

grubberr linked an issue Jun 7, 2022 that may be closed by this pull request

Stabilize source-github memory consumption to better scale with large streams #12863

Closed

github.md updated

e02ea63

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

octavia-squidington-iii added the connectors/source/github label Jun 7, 2022

grubberr changed the title ~~🐛 Source Github: cache only parent streams~~ 🐛 Source Github: Enable caching only parent streams Jun 7, 2022

grubberr changed the title ~~🐛 Source Github: Enable caching only parent streams~~ 🐛 Source Github: enable caching only parent streams Jun 7, 2022

github.md updated

4d89169

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr removed the area/documentation Improvements or additions to documentation label Jun 7, 2022

grubberr requested a review from ChristopheDuong June 7, 2022 15:43

grubberr mentioned this pull request Jun 7, 2022

Source Github: add incremental support to workflow streams #12266

Closed

grubberr requested review from lazebnyi, midavadim and Phlair and removed request for ChristopheDuong June 8, 2022 04:27

grubberr mentioned this pull request Jun 8, 2022

Source Github: Add requests-cache library to improve read execution of pull_request_stats stream #13264

Closed

lazebnyi approved these changes Jun 8, 2022

View reviewed changes

grubberr added 2 commits June 8, 2022 20:26

Merge branch 'master' into grubberr/12863-source-github

76d3daa

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github.md updated

19c69ac

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/documentation Improvements or additions to documentation label Jun 8, 2022

grubberr changed the title ~~🐛 Source Github: enable caching only parent streams~~ 🐛 Source Github: enable caching only for parent streams Jun 8, 2022

Merge branch 'master' into grubberr/12863-source-github

08b5b07

auto-bump connector version

3d079db

octavia-squidington-iii temporarily deployed to more-secrets June 9, 2022 04:04 Inactive

grubberr merged commit 9a4582a into master Jun 9, 2022

grubberr deleted the grubberr/12863-source-github branch June 9, 2022 04:21

octavia-squidington-iii mentioned this pull request Jun 9, 2022

Bump Airbyte version from 0.39.16-alpha to 0.39.17-alpha #13641

Merged

evantahler mentioned this pull request Jun 9, 2022

source-github should use smaller page sizes to prevent OOM issues #13629

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Github: enable caching only for parent streams #13558

🐛 Source Github: enable caching only for parent streams #13558

grubberr commented Jun 7, 2022 •

edited

Loading

codecov bot commented Jun 7, 2022 •

edited

Loading

grubberr commented Jun 7, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 8, 2022

grubberr commented Jun 8, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 8, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

🐛 Source Github: enable caching only for parent streams #13558

🐛 Source Github: enable caching only for parent streams #13558

Conversation

grubberr commented Jun 7, 2022 • edited Loading

What

workflow_runs stream VCR caching Enabled

workflow_runs stream VCR caching Disabled

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

codecov bot commented Jun 7, 2022 • edited Loading

Codecov Report

grubberr commented Jun 7, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Jun 8, 2022

grubberr commented Jun 8, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Jun 8, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Jun 9, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Jun 9, 2022 • edited by github-actions bot Loading

grubberr commented Jun 7, 2022 •

edited

Loading

codecov bot commented Jun 7, 2022 •

edited

Loading

grubberr commented Jun 7, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 8, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 8, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading