Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic end-to-end integration test #108

Merged
merged 5 commits into from
May 10, 2023
Merged

Add basic end-to-end integration test #108

merged 5 commits into from
May 10, 2023

Conversation

mwylde
Copy link
Member

@mwylde mwylde commented May 9, 2023

This PR adds a simple integration test that:

  • Starts up the services
  • Creates a nexmark source
  • Starts a pipeline with this SQL:
select count(*) from nexmark where auction is not null group
                by hop(interval '2 seconds', interval '10 seconds')
  • waits for the pipeline to get to the RUNNING state
  • waits for 10 successful checkpoints
  • stops the job
  • waits for it to successfully finish

Now that we have the ability to write integration tests, we can extend this:

  • More cases (stop and re-starts, rescaling, recovery, etc.)
  • More complex SQL
  • Sources and sinks
  • Correctness testing on output

This PR also changes the CI to use debug mode to speed it up a bit. Since we're not using the artifacts produced by it at all, doing everything in release mode isn't necessary.

config: Some(create_pipeline_req::Config::Sql(
arroyo_rpc::grpc::api::CreateSqlJob {
query: format!(
"select count(*) from {} where auction is not null group \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be nice to factor this out into a test-case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, eventually this will be extended into more of a test framework but for now I just wanted a basic test that validates we can compile and run a pipeline

@mwylde mwylde merged commit 11375ca into master May 10, 2023
@mwylde mwylde deleted the e2e branch May 10, 2023 01:18
edmondop pushed a commit to edmondop/arroyo that referenced this pull request May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants