roachtest: allow roachtests to opt out of specific metamorphic constants #113164
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
O-roachtest
T-testeng
TestEng Team
As of #111949, roachtests are now randomly run with runtime assertions and metamorphic constants enabled. Some roachtests make assumptions about the invariance of certain metamorphic constants.
One example is the
asyncpg
test which expectsmultiple_active_portals_enabled=false
and expects failures on tests that try and use multiple portals. However, if metamorphic constants are enabled, it is possible for multiple portals to be enabled and those tests will unexpectedly pass.Currently to fix this, we opt out those roachtest of metamorphic constants and runtime assertions completely by using the standard binary.
https://github.com/cockroachdb/cockroach/blob/1d6e2db725167afd308dc96c2d57e4bd27246064/pkg/cmd/roachtest/tests/split.go#L430
However, this is just a temporary fix as we still want to run these tests with metamorphic constants and runtime assertions. Instead, we should allow tests to specify a metamorphic constants exclusions list. If a constant is specified, that metamorphic constant should be disabled for that test only. This would let us disable metamorphic constants that break tests on a test by test basis.
One particular challenge to this is the fact that metamorphic constants are evaluated on initialization of the cockroach binary. This makes it non trivial to preemptively disable them from the test runner.
Jira issue: CRDB-32784
The text was updated successfully, but these errors were encountered: