Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/sql/logictest/tests/cockroach-go-testserver-24.3/cockroach-go-testserver-24_3_test: TestLogic_upgrade failed #139405

Closed
cockroach-teamcity opened this issue Jan 18, 2025 · 3 comments · Fixed by #140058
Closed

pkg/sql/logictest/tests/cockroach-go-testserver-24.3/cockroach-go-testserver-24_3_test: TestLogic_upgrade failed #139405

cockroach-teamcity opened this issue Jan 18, 2025 · 3 comments · Fixed by #140058
Assignees
@fqazi
Labels
branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 branch-release-25.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 18, 2025 •
edited by cockroach-jira-scripts
Loading

pkg/sql/logictest/tests/cockroach-go-testserver-24.3/cockroach-go-testserver-24_3_test.TestLogic_upgrade failed on release-25.1 @ 29c0a3d5161b6818e414efc9c35321c1c0416e19:

Fatal error:

panic: test timed out after 15m0s
running tests:
	TestLogic_upgrade (15m0s)

Stack:

goroutine 92 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

=== RUN   TestLogic_upgrade
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestLogic_upgrade1286049234
    test_log_scope.go:76: use -show-logs to present logs inline

Parameters:

  • attempt=1
  • run=5
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-46634
@cockroach-teamcity cockroach-teamcity added branch-release-25.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jan 18, 2025
@fqazi
Copy link
Collaborator

fqazi commented Jan 21, 2025

This looks similar to #139405

@exalate-issue-sync exalate-issue-sync bot added P-1 Issues/test failures with a fix SLA of 1 month GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 21, 2025
@cockroach-teamcity
Copy link
Member Author

pkg/sql/logictest/tests/cockroach-go-testserver-24.3/cockroach-go-testserver-24_3_test.TestLogic_upgrade failed on release-25.1 @ 4bb718e70117615cd5ac3210ae9dcfcd67097ab7:

Fatal error:

panic: test timed out after 15m0s
running tests:
	TestLogic_upgrade (15m0s)

Stack:

goroutine 115 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

=== RUN   TestLogic_upgrade
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestLogic_upgrade2180664852
    test_log_scope.go:76: use -show-logs to present logs inline

Parameters:

  • attempt=1
  • run=10
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

fqazi added a commit to fqazi/cockroach that referenced this issue Jan 29, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
588c9c1 
Previusly, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: cockroachdb#139837
Fixes: cockroachdb#139405
Fixes: cockroachdb#139356
Fixes: cockroachdb#139100

Release note: None
@fqazi fqazi mentioned this issue Jan 29, 2025
Merged
fqazi added a commit to fqazi/cockroach that referenced this issue Jan 29, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
6c7d979 
Previusly, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: cockroachdb#139837
Fixes: cockroachdb#139405
Fixes: cockroachdb#139356
Fixes: cockroachdb#139100

Release note: None
fqazi added a commit to fqazi/cockroach that referenced this issue Jan 30, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
db15359 
Previusly, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: cockroachdb#139837
Fixes: cockroachdb#139405
Fixes: cockroachdb#139356
Fixes: cockroachdb#139100

Release note: None
fqazi added a commit to fqazi/cockroach that referenced this issue Jan 30, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
d01312a 
Previusly, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: cockroachdb#139837
Fixes: cockroachdb#139405
Fixes: cockroachdb#139356
Fixes: cockroachdb#139100

Release note: None
fqazi added a commit to fqazi/cockroach that referenced this issue Jan 30, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
7c33794 
Previusly, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: cockroachdb#139837
Fixes: cockroachdb#139405
Fixes: cockroachdb#139356
Fixes: cockroachdb#139100

Release note: None
craig bot pushed a commit that referenced this issue Jan 30, 2025
@fqazi
Merge #140058
3f1b094 
140058: catalog/lease: disallow leases until range feed starts r=fqazi a=fqazi

Previusly, the lease manager allowed descriptors to be leased before the range feed was fully initialized. This was problematic because if an descriptor was updated before the range feed is started we would have no way of knowing. For example an early query during startup could fetch the system database descriptor before the range feed starts, and this could kept forever. If an upgrade bumped the descriptor version between the initial fetch and start of range feed then we could end up with a node stuck with the sale version forever. To address this, this patch forces any early lease acqusitions to wait for the range feed to start up.

Fixes: #139837
Fixes: #139405
Fixes: #139356
Fixes: #139100 
Informs: #139837
Informs: #139405 
Informs: #139356


Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
craig bot pushed a commit that referenced this issue Jan 31, 2025
@fqazi
Merge #140058
60c54cf 
140058: catalog/lease: disallow leases until range feed starts r=fqazi a=fqazi

Previusly, the lease manager allowed descriptors to be leased before the range feed was fully initialized. This was problematic because if an descriptor was updated before the range feed is started we would have no way of knowing. For example an early query during startup could fetch the system database descriptor before the range feed starts, and this could kept forever. If an upgrade bumped the descriptor version between the initial fetch and start of range feed then we could end up with a node stuck with the sale version forever. To address this, this patch forces any early lease acqusitions to wait for the range feed to start up.

Fixes: #139837
Fixes: #139405
Fixes: #139356
Fixes: #139100 
Informs: #139837
Informs: #139405 
Informs: #139356


Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
craig bot pushed a commit that referenced this issue Jan 31, 2025
@fqazi
Merge #140058
5b93185 
140058: catalog/lease: disallow leases until range feed starts r=fqazi a=fqazi

Previusly, the lease manager allowed descriptors to be leased before the range feed was fully initialized. This was problematic because if an descriptor was updated before the range feed is started we would have no way of knowing. For example an early query during startup could fetch the system database descriptor before the range feed starts, and this could kept forever. If an upgrade bumped the descriptor version between the initial fetch and start of range feed then we could end up with a node stuck with the sale version forever. To address this, this patch forces any early lease acqusitions to wait for the range feed to start up.

Fixes: #139837
Fixes: #139405
Fixes: #139356
Fixes: #139100 
Informs: #139837
Informs: #139405 
Informs: #139356


Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
@craig craig bot closed this as completed in 30a64a6 Jan 31, 2025
blathers-crl bot pushed a commit that referenced this issue Jan 31, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
18be106 
Previusly, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: #139837
Fixes: #139405
Fixes: #139356
Fixes: #139100

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Jan 31, 2025
Merged
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Feb 3, 2025

Based on the specified backports for linked PR #140058, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added the branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 label Feb 3, 2025
fqazi added a commit to fqazi/cockroach that referenced this issue Feb 3, 2025
@fqazi
catalog/lease: disallow leases until range feed starts
98aaded 
Previously, the lease manager allowed descriptors to be leased before the
range feed was fully initialized. This was problematic because if an
descriptor was updated before the range feed is started we would have no
way of knowing. For example an early query during startup could fetch
the system database descriptor before the range feed starts, and this
could kept forever. If an upgrade bumped the descriptor version between
the initial fetch and start of range feed then we could end up with a
node stuck with the sale version forever. To address this, this patch
forces any early lease acqusitions to wait for the range feed to start
up.

Fixes: cockroachdb#139837
Fixes: cockroachdb#139405
Fixes: cockroachdb#139356
Fixes: cockroachdb#139100

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 branch-release-25.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

catalog/lease: disallow leases until range feed starts fqazi/cockroach
2 participants
@fqazi @cockroach-teamcity