Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test: Add script to streamline integ test with any source #58

Merged
merged 4 commits into from
Feb 21, 2024

Conversation

aaronsteers
Copy link
Contributor

@aaronsteers aaronsteers commented Feb 21, 2024

This is not yet tested, and needs a bit more work.

This is working now. 😄

The goal here is to be able to run a sync using any source by running the script and passing the name of the source you want to test:

poetry run python examples/run_integ_test_source.py SOURCE_NAME

Or to test a specific stream:

poetry run python examples/run_integ_test_source.py SOURCE_NAME STREAMS_CSV

If provided, the second STREAMS_CSV arg is expected to be a comma-delimited string (not a file) with one or more streams. For instance: events (with no comma) or events,customers,users.

For instance, here are the commands I used for testing Source Klaviyo:

# Run all streams
poetry run python examples/run_integ_test_source.py source-klaviyo

# Run just the events stream
poetry run python examples/run_integ_test_source.py source-klaviyo events

@aaronsteers
Copy link
Contributor Author

First intended application is to try to repro and diagnose this issue:

@aaronsteers aaronsteers marked this pull request as ready for review February 21, 2024 19:03
@aaronsteers aaronsteers changed the title Tests: add script to streamline integ test with any source Test: add script to streamline integ test with any source Feb 21, 2024
@aaronsteers aaronsteers changed the title Test: add script to streamline integ test with any source Test: Add script to streamline integ test with any source Feb 21, 2024
@aaronsteers
Copy link
Contributor Author

Merging, as this appears to be working well now. I expect we will continue to iterate and improve as we go, for instance, by looping through streams one at a time, and perhaps running additional debug checks to narrow down the issue when a failure occurs.

@aaronsteers aaronsteers merged commit b0a98fe into main Feb 21, 2024
9 of 12 checks passed
@aaronsteers aaronsteers deleted the aj/streamline-integ-tests branch February 21, 2024 20:41
@aaronsteers aaronsteers self-assigned this Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant