Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-stripe: Add missing fields to invoices and subscriptions schemas and improve memory usage when reading from substreams #34347

Merged
merged 18 commits into from
Jan 19, 2024
Merged

source-stripe: Add missing fields to invoices and subscriptions schemas and improve memory usage when reading from substreams #34347

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8ac8d14
bump cdk version
girarda Jan 18, 2024
23b572a
remove extra print
girarda Jan 18, 2024
8271eb5
Update stripe.md
girarda Jan 18, 2024
7a99c7d
bump
girarda Jan 18, 2024
9539085
Merge branch 'alex/stripe_bump_cdk' of github.com:airbytehq/airbyte i…
girarda Jan 18, 2024
3dfe35c
Merge branch 'master' into alex/stripe_bump_cdk
girarda Jan 18, 2024
ea728cb
Merge branch 'master' into alex/stripe_bump_cdk
girarda Jan 18, 2024
ef36ba3
update expected_records
girarda Jan 18, 2024
344424f
add issuer
girarda Jan 18, 2024
d7bf441
add issuer field to schema
girarda Jan 19, 2024
f3f9e54
move
girarda Jan 19, 2024
8d3da4a
update subscriptions record
girarda Jan 19, 2024
00b2c5e
update expected records
girarda Jan 19, 2024
54973dd
Revert "update expected records"
girarda Jan 19, 2024
90ebe3b
update invoice record
girarda Jan 19, 2024
5421856
update the other invoices
girarda Jan 19, 2024
e1d7187
update another record
girarda Jan 19, 2024
261427d
Upgrade to minor bump
girarda Jan 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions airbyte-integrations/connectors/source-stripe/integration_tests/expected_records.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-stripe/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: e094cb9a-26de-4645-8761-65c0c425d1de
dockerImageTag: 5.1.3
dockerImageTag: 5.2.0
dockerRepository: airbyte/source-stripe
documentationUrl: https://docs.airbyte.com/integrations/sources/stripe
githubIssueLabel: source-stripe
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-stripe/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

from setuptools import find_packages, setup

MAIN_REQUIREMENTS = ["airbyte-cdk==0.57.6", "stripe==2.56.0", "pendulum==2.1.2"]
MAIN_REQUIREMENTS = ["airbyte-cdk==0.58.9", "stripe==2.56.0", "pendulum==2.1.2"]

# we set `requests-mock~=1.11.0` to ensure concurrency is supported
TEST_REQUIREMENTS = ["pytest-mock~=3.6.1", "pytest~=6.1", "requests-mock~=1.11.0", "requests_mock~=1.8", "freezegun==1.2.2"]
Expand Down
8 changes: 8 additions & 0 deletions airbyte-integrations/connectors/source-stripe/source_stripe/schemas/invoices.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,14 @@
}
}
},
"issuer": {
"type": ["null", "object"],
"properties": {
"type": {
"type": ["null", "string"]
}
}
},
"latest_revision": {
"type": ["null", "string"]
},
Expand Down
6 changes: 6 additions & 0 deletions airbyte-integrations/connectors/source-stripe/source_stripe/schemas/subscriptions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@
"billing_cycle_anchor": {
"type": ["null", "number"]
},
"billing_cycle_anchor_config": {
"type": ["null", "object"]
},
"invoice_settings": {
"type": ["null", "object"]
},
"cancel_at_period_end": {
"type": ["null", "boolean"]
},
Expand Down
14 changes: 0 additions & 14 deletions airbyte-integrations/connectors/source-stripe/source_stripe/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -707,20 +707,6 @@ def read_records(self, sync_mode: SyncMode, stream_slice: Optional[Mapping[str,
items_next_pages = super().read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice, **kwargs)
yield from chain(items, items_next_pages)

def stream_slices(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@darynaishchenko can you confirm this method was only added for logging purposes as part of this PR?

This is creating a lot of log entries (one per parent record), so I suggest we revert this

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@girarda yes, that is only for logs purpose

self, sync_mode: SyncMode, cursor_field: Optional[List[str]] = None, stream_state: Optional[Mapping[str, Any]] = None
) -> Iterable[Optional[Mapping[str, Any]]]:
parent_stream_slices = self.parent.stream_slices(
sync_mode=SyncMode.full_refresh, cursor_field=cursor_field, stream_state=stream_state
)
for stream_slice in parent_stream_slices:
parent_records = self.parent.read_records(
sync_mode=SyncMode.full_refresh, cursor_field=cursor_field, stream_slice=stream_slice, stream_state=stream_state
)
for record in parent_records:
self.logger.info(f"Fetching parent stream slices for stream {self.name}.")
yield {"parent": record}


class IncrementalStripeLazySubStreamSelector(IStreamSelector):
def __init__(self, updated_cursor_incremental_stream: UpdatedCursorIncrementalStripeStream, lazy_sub_stream: StripeLazySubStream):
Expand Down
5 changes: 3 additions & 2 deletions docs/integrations/sources/stripe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,9 +216,10 @@ Each record is marked with `is_deleted` flag when the appropriate event happens

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 5.2.0 | 2024-01-18 | [34347](https://github.com/airbytehq/airbyte/pull//34347) | Add new fields invoices and subscription streams. Upgrade the CDK for better memory usage. |
| 5.1.3 | 2023-12-18 | [33306](https://github.com/airbytehq/airbyte/pull/33306/) | Adding integration tests |
| 5.1.2 | 2024-01-04 | [33414](https://github.com/airbytehq/airbyte/pull/33414) | Prepare for airbyte-lib |
| 5.1.1 | 2024-01-04 | [33926](https://github.com/airbytehq/airbyte/pull/33926/) | Update endpoint for `bank_accounts` stream |
| 5.1.2 | 2024-01-04 | [33414](https://github.com/airbytehq/airbyte/pull/33414) | Prepare for airbyte-lib |
| 5.1.1 | 2024-01-04 | [33926](https://github.com/airbytehq/airbyte/pull/33926/) | Update endpoint for `bank_accounts` stream |
| 5.1.0 | 2023-12-11 | [32908](https://github.com/airbytehq/airbyte/pull/32908/) | Read full refresh streams concurrently |
| 5.0.2 | 2023-12-01 | [33038](https://github.com/airbytehq/airbyte/pull/33038) | Add stream slice logging for SubStream |
| 5.0.1 | 2023-11-17 | [32638](https://github.com/airbytehq/airbyte/pull/32638/) | Availability stretegy: check availability of both endpoints (if applicable) - common API + events API |
Expand Down
Loading