Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: allow roachtests to opt out of specific metamorphic constants #113164

Open
DarrylWong opened this issue Oct 26, 2023 · 1 comment
Open
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-roachtest T-testeng TestEng Team

Comments

@DarrylWong
Copy link
Contributor

DarrylWong commented Oct 26, 2023

As of #111949, roachtests are now randomly run with runtime assertions and metamorphic constants enabled. Some roachtests make assumptions about the invariance of certain metamorphic constants.

One example is the asyncpg test which expects multiple_active_portals_enabled=false and expects failures on tests that try and use multiple portals. However, if metamorphic constants are enabled, it is possible for multiple portals to be enabled and those tests will unexpectedly pass.

Currently to fix this, we opt out those roachtest of metamorphic constants and runtime assertions completely by using the standard binary.

https://github.com/cockroachdb/cockroach/blob/1d6e2db725167afd308dc96c2d57e4bd27246064/pkg/cmd/roachtest/tests/split.go#L430

However, this is just a temporary fix as we still want to run these tests with metamorphic constants and runtime assertions. Instead, we should allow tests to specify a metamorphic constants exclusions list. If a constant is specified, that metamorphic constant should be disabled for that test only. This would let us disable metamorphic constants that break tests on a test by test basis.

One particular challenge to this is the fact that metamorphic constants are evaluated on initialization of the cockroach binary. This makes it non trivial to preemptively disable them from the test runner.

Jira issue: CRDB-32784

@DarrylWong DarrylWong added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-roachtest T-testeng TestEng Team labels Oct 26, 2023
@blathers-crl
Copy link

blathers-crl bot commented Oct 26, 2023

cc @cockroachdb/test-eng

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-roachtest T-testeng TestEng Team
Projects
No open projects
Status: Triage
Development

No branches or pull requests

1 participant