Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add standalone e2e test cases for deterministic recovery test #6047

Closed
yezizp2012 opened this issue Oct 26, 2022 · 2 comments · Fixed by #6252
Closed

test: add standalone e2e test cases for deterministic recovery test #6047

yezizp2012 opened this issue Oct 26, 2022 · 2 comments · Fixed by #6252
Labels
help wanted Issues that need help from contributors type/feature

Comments

@yezizp2012
Copy link
Member

yezizp2012 commented Oct 26, 2022

Is your feature request related to a problem? Please describe.

Currently we will run deterministic recovery test as follows for all PRs:

echo "--- deterministic simulation e2e, ci-3cn-1fe, recovery, streaming"
seq 1 | parallel MADSIM_TEST_SEED={} $RUNNER --kill-meta --kill-frontend --kill-compute './e2e_test/streaming/\*\*/\*.slt'

# bugs here! Tracking issue https://github.com/risingwavelabs/risingwave/issues/4527
echo "--- deterministic simulation e2e, ci-3cn-1fe, recovery, batch"
seq 1 | parallel MADSIM_TEST_SEED={} $RUNNER --kill-meta --kill-frontend --kill-compute './e2e_test/batch/\*\*/\*.slt'

But:

  1. there're so many test cases under ./e2e_test/streaming/, it will definitely timeout since the test will randomly kill components during running each sql except DML.
  2. the test is soft pass now because some bugs are still under fixing. This can lead to new bugs not being found when they appear, which has happened more than once.

Describe the solution you'd like

We should sort out some e2e test cases that recovery tests may involve, the cases should includes:

  1. ddl operations, including table/stream sources.
  2. cases that cover almost all Executors to test whether they will recovery properly.
  3. data consistency, including check of the ability to handle dml correctly after recovery.

After that, we should force enable the deterministic recovery test to check whether this PR will break any recovery process.

Describe alternatives you've considered

No response

Additional context

No response

@github-actions github-actions bot added this to the release-0.1.14 milestone Oct 26, 2022
@yezizp2012 yezizp2012 added the help wanted Issues that need help from contributors label Oct 26, 2022
@yezizp2012
Copy link
Member Author

Cc @wangrunji0408

@yezizp2012
Copy link
Member Author

Currently only dynamic_filter.slt, outer_join.slt and tpch_upstream.slt will fail the deterministic recovery test, the bugs are recorded in #5726 #6093 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Issues that need help from contributors type/feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant