Source Stripe: fix stream schemas #21858

roman-yermilov-gl · 2023-01-25T16:33:27Z

What

https://github.com/airbytehq/oncall/issues/1364

How

Fix stream schemas:

start -> start_date
discount.start_date -> discount.start
for subscription stream,
add status_transitions for invoices stream

roman-yermilov-gl · 2023-01-25T16:36:51Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4007987867
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4007987867
Python tests coverage:

Name                        Stmts   Miss  Cover
-----------------------------------------------
source_stripe/source.py        22      0   100%
source_stripe/__init__.py       2      0   100%
source_stripe/streams.py      304     27    91%
-----------------------------------------------
TOTAL                         328     27    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
============ 49 passed, 1 skipped, 31 warnings in 927.91s (0:15:27) ============

evantahler · 2023-01-25T16:49:48Z

airbyte-integrations/connectors/source-stripe/source_stripe/schemas/subscriptions.json

-    "start": {
+    "start_date": {


I'm interested to know why a failing test didn't catch this before? Is it because we have yet to test this stream with expected_records?

We can potentially test it with expected records if we strictly compare records. If not, then we allowed by default to have additional fields that not specified in schema. We currently have task to migrate to stripe tests to strict level (assigned to me) so I can check it during migration

We currently have task to migrate to stripe tests to strict level (assigned to me) so I can check it during migration

Does this imply that we intend to have all the fields in the schema, but we just missed this one? (And the future test work will help us catch it faster?)

I just have remembered that migration did not catch it properly with expected records :)

I just have remembered that migration did not catch it properly with expected records :)

Can you please file a bug about this then?

@evantahler @mfsiega-airbyte I'm not sure this is actually a bug. A missing field may be not spotted by the test because we guarantee that additionalProperties field always equals True. Do you think we should override this requirement when testing if data matches its schema? if yes, I will create an issue for that.
Another thing is that we have lots of streams not filled with data so the test is just skipped. This should be fixed for GA connectors when they all have a test strictness level set to high

mfsiega-airbyte · 2023-01-25T21:46:04Z

One question: the report here mentioned at least one other field missing (status_transitions in the Invoices stream). Shall we add that here? And more generally, do we have a mechanism to ensure that all fields are included? (I see a thread about testing?)

davydov-d · 2023-01-26T12:04:21Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4014953645

davydov-d · 2023-01-26T12:13:55Z

Bumped a major version because of schema changes

davydov-d · 2023-01-26T12:14:06Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4015028609
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4015028609
Python tests coverage:

Name                        Stmts   Miss  Cover
-----------------------------------------------
source_stripe/source.py        22      0   100%
source_stripe/__init__.py       2      0   100%
source_stripe/streams.py      304     27    91%
-----------------------------------------------
TOTAL                         328     27    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
============ 49 passed, 1 skipped, 31 warnings in 906.37s (0:15:06) ============

davydov-d · 2023-01-26T12:17:02Z

One question: the report here mentioned at least one other field missing (status_transitions in the Invoices stream). Shall we add that here? And more generally, do we have a mechanism to ensure that all fields are included? (I see a thread about testing?)

@mfsiega-airbyte I added status_transitions field within this PR. Regarding your question about a mechanism to ensure that all fields are included, please see my comment above. I think it is possible for GA connectors when they all have high test strictness level. Additionally, we should probably report a bug for source-acceptance-tests package (but I'm not sure about this)

davydov-d · 2023-01-26T12:55:58Z

/publish connector=connectors/source-stripe

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2023-01-26T12:59:26Z

/publish connector=connectors/source-stripe

🕑 Publishing the following connectors:
connectors/source-stripe
https://github.com/airbytehq/airbyte/actions/runs/4015357269

Connector	Did it publish?	Were definitions generated?
connectors/source-stripe	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2023-01-26T13:40:24Z

will merge this one when @klsoper approves

github-actions · 2023-01-26T13:44:14Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	23.97%	❌

Source Stripe: fix field name for subscription stream

2570b27

roman-yermilov-gl self-assigned this Jan 25, 2023

octavia-squidington-iv added area/connectors Connector related issues connectors/source/stripe labels Jan 25, 2023

Source Stripe: bump version

925cbd3

roman-yermilov-gl requested review from darynaishchenko, evantahler and lazebnyi January 25, 2023 16:34

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Jan 25, 2023

Source Stripe: update changelog

f92b0cf

evantahler reviewed Jan 25, 2023

View reviewed changes

roman-yermilov-gl requested a review from evantahler January 25, 2023 16:56

evantahler approved these changes Jan 25, 2023

View reviewed changes

lazebnyi approved these changes Jan 25, 2023

View reviewed changes

#1364 Source Stripe: fix stream schemas

6a2892e

davydov-d changed the title ~~Ryermilov/source stripe fix subscription schema~~ Source Stripe: fix stream schemas Jan 26, 2023

#1364 source Stripe: bump major version

6bd1291

auto-bump connector version

7754df7

octavia-squidington-iii temporarily deployed to more-secrets January 26, 2023 13:34 — with GitHub Actions Inactive

davydov-d merged commit 01b0565 into master Jan 27, 2023

davydov-d deleted the ryermilov/source-stripe-fix-subscription-schema branch January 27, 2023 18:10

octavia-squidington-iii mentioned this pull request Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Stripe: fix stream schemas #21858

Source Stripe: fix stream schemas #21858

roman-yermilov-gl commented Jan 25, 2023 •

edited by davydov-d

Loading

roman-yermilov-gl commented Jan 25, 2023 •

edited by github-actions bot

Loading

evantahler Jan 25, 2023

roman-yermilov-gl Jan 25, 2023

mfsiega-airbyte Jan 25, 2023

roman-yermilov-gl Jan 25, 2023

evantahler Jan 25, 2023

davydov-d Jan 26, 2023

mfsiega-airbyte commented Jan 25, 2023

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023

github-actions bot commented Jan 26, 2023

Source Stripe: fix stream schemas #21858

Source Stripe: fix stream schemas #21858

Conversation

roman-yermilov-gl commented Jan 25, 2023 • edited by davydov-d Loading

What

How

roman-yermilov-gl commented Jan 25, 2023 • edited by github-actions bot Loading

Build Passed

evantahler Jan 25, 2023

Choose a reason for hiding this comment

roman-yermilov-gl Jan 25, 2023

Choose a reason for hiding this comment

mfsiega-airbyte Jan 25, 2023

Choose a reason for hiding this comment

roman-yermilov-gl Jan 25, 2023

Choose a reason for hiding this comment

evantahler Jan 25, 2023

Choose a reason for hiding this comment

davydov-d Jan 26, 2023

Choose a reason for hiding this comment

mfsiega-airbyte commented Jan 25, 2023

davydov-d commented Jan 26, 2023 • edited by github-actions bot Loading

davydov-d commented Jan 26, 2023

davydov-d commented Jan 26, 2023 • edited by github-actions bot Loading

Build Passed

davydov-d commented Jan 26, 2023

davydov-d commented Jan 26, 2023 • edited by github-actions bot Loading

davydov-d commented Jan 26, 2023 • edited by github-actions bot Loading

davydov-d commented Jan 26, 2023

github-actions bot commented Jan 26, 2023

Airbyte Code Coverage

roman-yermilov-gl commented Jan 25, 2023 •

edited by davydov-d

Loading

roman-yermilov-gl commented Jan 25, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading

davydov-d commented Jan 26, 2023 •

edited by github-actions bot

Loading