Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

streamingccl: TestTenantStreamingDeleteRange is flaky #85630

Closed
stevendanna opened this issue Aug 4, 2022 · 2 comments · Fixed by #87532
Closed

streamingccl: TestTenantStreamingDeleteRange is flaky #85630

stevendanna opened this issue Aug 4, 2022 · 2 comments · Fixed by #87532
Assignees
Labels
A-tenant-streaming Including cluster streaming C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. skipped-test T-multitenant Issues owned by the multi-tenant virtual team
Milestone

Comments

@stevendanna
Copy link
Collaborator

stevendanna commented Aug 4, 2022

Describe the problem

The test fails quickly under stress:

./dev test ./pkg/ccl/streamingccl/streamingest --filter TestTenantStreamingDeleteRange --stress

Jira issue: CRDB-18349

Epic CRDB-27931

@stevendanna stevendanna added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Aug 4, 2022
stevendanna added a commit to stevendanna/cockroach that referenced this issue Aug 4, 2022
Refs: cockroachdb#85630

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
craig bot pushed a commit that referenced this issue Aug 4, 2022
85578: cast: remove unused function call r=yuzefovich a=yuzefovich

This was introduced in 7d0f75a, seems
like a leftover from debugging.

Release note: None

85628: bazel: bump size of `streamingest_test` r=jlinder a=rickystewart

This has timed out in CI.

Release note: None

85631: ccl/streamingccl/streamingest: skip TestTenantStreamingDeleteRange r=adityamaru a=stevendanna

Refs: #85630

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None

Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Steven Danna <danna@cockroachlabs.com>
stevendanna added a commit to stevendanna/cockroach that referenced this issue Aug 9, 2022
In projectSetProcessor, the `gen` array is not populated until
nextInputRow() is called in Next.  But, we expected that these
generators had been populated in MustBeStreaming which is called
before Next() is called.

As a result, rows returned streaming value generators were still
buffered.

I believe this is the main cause of cockroachdb#85630 and other odd behaviour
we've seen in the tenant replication tests.

Release note: None
@knz knz added the A-tenant-streaming Including cluster streaming label Sep 6, 2022
@blathers-crl
Copy link

blathers-crl bot commented Sep 6, 2022

cc @cockroachdb/tenant-streaming

@knz knz added T-multitenant Issues owned by the multi-tenant virtual team skipped-test labels Sep 6, 2022
@knz knz added this to the 22.2 milestone Sep 6, 2022
@adityamaru adityamaru self-assigned this Sep 6, 2022
@adityamaru
Copy link
Contributor

Under stress I've gone 5+ mins without a failure I suspect #85866 fixed this and we just forgot to unskip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-tenant-streaming Including cluster streaming C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. skipped-test T-multitenant Issues owned by the multi-tenant virtual team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants