Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

increase max offset for crdb cluster in e2e tests #389

Merged
merged 1 commit into from
Jan 28, 2022

Conversation

ecordell
Copy link
Contributor

@ecordell ecordell commented Jan 27, 2022

analyzing recent e2e flakes suggested that:

  • if the gha runner is slower (that it was when the test was written), the clock skew may be enough to mark a node as bad and be omitted from the cockroach cluster (thus ruining the test)
  • write performance on the gha runners seems to have dropped, so that the steps that fill crdb with data sometimes stall out

I bumped the max-offset for the cluster which should prevent crdb nodes from being marked bad even with the offsets we use to demonstrate the new enemy problem.

I also reduced the amount of data that the test writes - it was overly aggressive before just because (at the time) it was still faster than the other test suites. This should™️ prevent the test from stalling out on writing filler data.

I re-ran e2e twice on this PR after these changes and both times they converged very quickly.

@github-actions github-actions bot added the area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) label Jan 28, 2022
@ecordell ecordell enabled auto-merge January 28, 2022 14:50
@ecordell ecordell merged commit 4abff41 into authzed:main Jan 28, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Jan 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants