Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: run nightlies with assertions enabled #128841

Closed
3 tasks
DarrylWong opened this issue Aug 12, 2024 · 1 comment · Fixed by #128699
Closed
3 tasks

roachtest: run nightlies with assertions enabled #128841

DarrylWong opened this issue Aug 12, 2024 · 1 comment · Fixed by #128699
Assignees
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-testeng TestEng Team

Comments

@DarrylWong
Copy link
Contributor

DarrylWong commented Aug 12, 2024

In #111949 we briefly enabled roachtest nightlies to run with the crdb_test flag on. This enabled roachtests to be run with runtime assertions as well as metamorphic constants. However, due to the "metamorphic" nature of metamorphic constants, it was difficult to test all possible combinations/constants and tests frequently flaked. To reduce noise, we disabled both runtime assertions and metamorphic constants in #114618.

Runtime assertions however, seemed to be more stable. In terms of infra flakes, we expect they might contribute to longer run times, but other failures are likely actual assertions hit or issues with the assertion themselves. We should look into reenabling just runtime assertions on roachtest nightlies.

Runtime assertions and metamorphic constants are both enabled by default when building with --crdb_test. To disable metamorphic constants, COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=true should be passed in when the cluster is started.

Related work/issues also includes:

  • The roachprod/roachtest framework uses metamorphic testing/constants and runtime assertions interchangeably. Now that we are only running with runtime assertions, we should make a clearer distinction.
  • The metamorphic constant framework unconditionally logs constants on init. Passing in COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=true on cluster startup is sufficient to disable constants in the cluster, but it won't stop logging on every cockroach binary invocation, i.e. ./cockroach workload
  • Since we suspect this might lead to longer run times, we should route timeouts to test-eng while we let this change stabalize.

Jira issue: CRDB-41227

@DarrylWong DarrylWong added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Aug 12, 2024
@DarrylWong DarrylWong self-assigned this Aug 12, 2024
@craig craig bot closed this as completed in 5897903 Aug 21, 2024
@exalate-issue-sync exalate-issue-sync bot added the T-testeng TestEng Team label Aug 30, 2024
Copy link

blathers-crl bot commented Aug 30, 2024

cc @cockroachdb/test-eng

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-testeng TestEng Team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant