Source Recharge: skip stream if 403 received #17608

arsenlosenko · 2022-10-05T13:44:43Z

What

Resolving:
https://github.com/airbytehq/alpha-beta-issues/issues/289

How

Problem in the issue is caused by lack of permissions to retrieve a specific resource, so in order to handle this issue gracefully I have added a condition which will skip the problematic stream and continue the sync itself.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

arsenlosenko · 2022-10-05T13:57:10Z

/test connector=connectors/source-recharge

🕑 connectors/source-recharge https://github.com/airbytehq/airbyte/actions/runs/3190323418
✅ connectors/source-recharge https://github.com/airbytehq/airbyte/actions/runs/3190323418
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_recharge/source.py        23      0   100%
source_recharge/__init__.py       2      0   100%
source_recharge/api.py           85      8    91%
-------------------------------------------------
TOTAL                           110      8    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

CLAassistant · 2022-10-05T14:17:51Z

All committers have signed the CLA.

lazebnyi

Small comments, other looks good.

airbyte-integrations/connectors/source-recharge/source_recharge/api.py

docs/integrations/sources/recharge.md

arsenlosenko · 2022-10-05T16:19:40Z

/publish connector=connectors/source-recharge

🕑 Publishing the following connectors:
connectors/source-recharge
https://github.com/airbytehq/airbyte/actions/runs/3191238942

Connector	Did it publish?	Were definitions generated?
connectors/source-recharge	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

arsenlosenko · 2022-10-05T16:26:26Z

/test connector=connectors/source-recharge

🕑 connectors/source-recharge https://github.com/airbytehq/airbyte/actions/runs/3191242840
✅ connectors/source-recharge https://github.com/airbytehq/airbyte/actions/runs/3191242840
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_recharge/source.py        23      0   100%
source_recharge/__init__.py       2      0   100%
source_recharge/api.py           86      8    91%
-------------------------------------------------
TOTAL                           111      8    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

…vation * master: (26 commits) supply a source id for schema discovery in connector integration tests (#17662) Source Iterable: Add permission check for stream (#17602) Moving TrackingClientSingleton.initialize into the bean itself (#17631) Handle null workspace IDs in tracking/reporting methods gracefully (#17641) Bump Airbyte version from 0.40.11 to 0.40.12 (#17653) Revert "Do not wait the end of a reset to return an update (#17591)" (#17640) Standardize HttpRequester's url_base and path format (#17524) Create geography_type enum and add geography column in connection and workspace table (#16818) airbyte-cron: update connector definitions from remote (#16438) Do not wait the end of a reset to return an update (#17591) Remove redundant title labels from connector specs (#17544) Updated GA4 status support large schema discovery (#17394) 🪟 🐛 Fixes connector checks not properly ending their loading state (#17620) 🪟🧪 [Experiment] add hideOnboarding experiment (#17605) Source Recharge: change releaseStage to GA (#17606) Source Recharge: skip stream if 403 received (#17608) remove sonar-scan workflow (#17609) Mark/tables should be full width on all pages (#17401) Auto fail all workfow if there is a Versioning issue (#17562) ...

* Source Recharge: skip stream if 403 received * Refactor should_retry method * Add logging message for clarity * Update changelog * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Source Recharge: skip stream if 403 received

1ee7d72

arsenlosenko requested review from grubberr and lazebnyi October 5, 2022 13:44

arsenlosenko self-assigned this Oct 5, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 5, 2022

octavia-squidington-iv added the connectors/source/recharge label Oct 5, 2022

Refactor should_retry method

7b2b1fe

lazebnyi approved these changes Oct 5, 2022

View reviewed changes

airbyte-integrations/connectors/source-recharge/source_recharge/api.py Show resolved Hide resolved

docs/integrations/sources/recharge.md Outdated Show resolved Hide resolved

Arsen Losenko added 2 commits October 5, 2022 19:18

Add logging message for clarity

23a7e60

Update changelog

8776a8f

auto-bump connector version [ci skip]

2debd66

arsenlosenko merged commit 99cdb40 into master Oct 5, 2022

arsenlosenko deleted the arsenlosenko/289-source-recharge-403-handling branch October 5, 2022 17:26

This was referenced Oct 5, 2022

Bump Airbyte version from 0.40.11 to 0.40.12 #17635

Closed

Bump Airbyte version from 0.40.11 to 0.40.12 #17653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Recharge: skip stream if 403 received #17608

Source Recharge: skip stream if 403 received #17608

arsenlosenko commented Oct 5, 2022 •

edited

Loading

arsenlosenko commented Oct 5, 2022 •

edited by github-actions bot

Loading

CLAassistant commented Oct 5, 2022 •

edited

Loading

lazebnyi left a comment

arsenlosenko commented Oct 5, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Oct 5, 2022 •

edited by github-actions bot

Loading

Source Recharge: skip stream if 403 received #17608

Source Recharge: skip stream if 403 received #17608

Conversation

arsenlosenko commented Oct 5, 2022 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

arsenlosenko commented Oct 5, 2022 • edited by github-actions bot Loading

Build Passed

CLAassistant commented Oct 5, 2022 • edited Loading

lazebnyi left a comment

Choose a reason for hiding this comment

arsenlosenko commented Oct 5, 2022 • edited by github-actions bot Loading

arsenlosenko commented Oct 5, 2022 • edited by github-actions bot Loading

Build Passed

arsenlosenko commented Oct 5, 2022 •

edited

Loading

arsenlosenko commented Oct 5, 2022 •

edited by github-actions bot

Loading

CLAassistant commented Oct 5, 2022 •

edited

Loading

arsenlosenko commented Oct 5, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Oct 5, 2022 •

edited by github-actions bot

Loading