Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ source-stripe: read multiple streams concurrently #32438

Closed
wants to merge 79 commits into from
Closed
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
95230db
Apply log level at the entrpoint level instead of the abstract source
girarda Nov 10, 2023
fee9d5a
remove apply logging from declarative framework
girarda Nov 10, 2023
b49ddbd
fix mypy
girarda Nov 10, 2023
c714cd6
apply changes
girarda Nov 10, 2023
84c6b50
pass airbytestream instead of configured stream to as_airbyte_message
girarda Nov 10, 2023
6275446
Merge branch 'alex/stream_status' into alex/concurrent_source
girarda Nov 10, 2023
18d87cd
update usage
girarda Nov 10, 2023
18dc14a
oops
girarda Nov 10, 2023
149212a
update
girarda Nov 10, 2023
03f53d6
rename
girarda Nov 10, 2023
e25f320
delte comment
girarda Nov 10, 2023
551a812
delete some dead code
girarda Nov 10, 2023
72fd3de
unit tests
girarda Nov 10, 2023
c7ea97a
start extracting into queue item handler
girarda Nov 10, 2023
a7a822a
move handle partition
girarda Nov 10, 2023
1a76377
on_partition_complete_sentinel
girarda Nov 10, 2023
2f7878f
delete dead code
girarda Nov 10, 2023
798fda7
delete more code
girarda Nov 10, 2023
438a0ea
on record
girarda Nov 10, 2023
1c3cd1c
on exception
girarda Nov 10, 2023
c918082
delete dead code
girarda Nov 10, 2023
2704468
is_done
girarda Nov 10, 2023
b6596f7
remove unused params
girarda Nov 10, 2023
2056d25
update
girarda Nov 10, 2023
d8ef071
fix
girarda Nov 10, 2023
4157008
delete dead code
girarda Nov 10, 2023
9a68f78
fix mypy
girarda Nov 10, 2023
dbe9262
cleanup the tests a bit
girarda Nov 10, 2023
907755a
update
girarda Nov 10, 2023
11ab634
remove param
girarda Nov 10, 2023
0a7c3f2
set max number of streams
girarda Nov 10, 2023
acd78b8
update
girarda Nov 10, 2023
39955e7
update
girarda Nov 10, 2023
5185868
update
girarda Nov 10, 2023
5b9fa3d
delete cruft
girarda Nov 10, 2023
0600795
delete cruft
girarda Nov 10, 2023
9254d55
update comment
girarda Nov 10, 2023
29ee92f
type annotation
girarda Nov 10, 2023
7dc764c
update
girarda Nov 10, 2023
e86b701
comments
girarda Nov 10, 2023
6843bfc
rename
girarda Nov 10, 2023
8ed60df
remove superfluous param
girarda Nov 10, 2023
c5a5e99
comment
girarda Nov 10, 2023
60d1a74
docstrings
girarda Nov 10, 2023
6ddea7f
small cleanup
girarda Nov 10, 2023
d2c3f86
rename
girarda Nov 10, 2023
d758fae
small update
girarda Nov 10, 2023
22aa854
docstrign
girarda Nov 10, 2023
7c8db6a
docstrings
girarda Nov 10, 2023
6bfed39
docstring
girarda Nov 10, 2023
6b56ee8
docstring
girarda Nov 10, 2023
a385b93
rename file
girarda Nov 10, 2023
6e6251c
use concurrent source
girarda Nov 10, 2023
2b89ac9
mypy
girarda Nov 10, 2023
2991317
Merge branch 'alex/concurrent_source' into alex/stripe_concurrent_str…
girarda Nov 10, 2023
328b763
update setup
girarda Nov 10, 2023
cdc0d8f
remove superfluous call to shutdown
girarda Nov 13, 2023
6b8d34e
add unit test
girarda Nov 13, 2023
55203ec
yield from []
girarda Nov 13, 2023
63199fc
remove useless comment
girarda Nov 13, 2023
e2fe651
comment
girarda Nov 13, 2023
c84d150
fix comment
girarda Nov 13, 2023
623bdd9
remove superfluous check
girarda Nov 13, 2023
e7a0493
extract a facade
girarda Nov 14, 2023
cae8795
test the adapter
girarda Nov 14, 2023
53c3c7e
rename field and comment
girarda Nov 14, 2023
700bb68
rename field
girarda Nov 14, 2023
927e1bc
mypy
girarda Nov 14, 2023
9a1c26e
rename
girarda Nov 14, 2023
df0d2ea
comments
girarda Nov 14, 2023
65cf274
fix tests
girarda Nov 14, 2023
64faf27
Merge branch 'alex/concurrent_source' into alex/stripe_concurrent_str…
girarda Nov 14, 2023
e053d19
update
girarda Nov 14, 2023
fd68c3e
use spec
girarda Nov 14, 2023
39a5f48
use public interface
girarda Nov 14, 2023
166d004
remove message repository property
girarda Nov 14, 2023
7789eb5
Merge branch 'alex/concurrent_source' into alex/stripe_concurrent_str…
girarda Nov 14, 2023
886feb1
Merge branch 'master' into alex/stripe_concurrent_streams
girarda Nov 28, 2023
016c9eb
delete dead code
girarda Nov 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,6 @@ def __init__(self, catalog: Optional[ConfiguredAirbyteCatalog], **kwargs):
# things will NOT be executed concurrently
self._streams_configured_as_full_refresh = set()

message_repository = InMemoryMessageRepository(entrypoint_logger.level)

@staticmethod
def validate_and_fill_with_defaults(config: MutableMapping) -> MutableMapping:
start_date, lookback_window_days, slice_range = (
Expand All @@ -77,7 +75,8 @@ def validate_and_fill_with_defaults(config: MutableMapping) -> MutableMapping:
)
if start_date:
try:
start_date = pendulum.parse(start_date).int_timestamp
if not isinstance(start_date, int):
start_date = pendulum.parse(start_date).int_timestamp
except pendulum.parsing.exceptions.ParserError as e:
message = f"Invalid start date {start_date}. Please use YYYY-MM-DDTHH:MM:SSZ format."
raise AirbyteTracedException(
Expand Down
Loading