Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaking TestLeaseGrantTimeToLiveExpired #17504

Open
ahrtr opened this issue Feb 28, 2024 · 5 comments
Open

Flaking TestLeaseGrantTimeToLiveExpired #17504

ahrtr opened this issue Feb 28, 2024 · 5 comments
Assignees
@fuweid
Labels
area/testing help wanted type/flake

Comments

@ahrtr
Copy link
Member

ahrtr commented Feb 28, 2024

Which Github Action / Prow Jobs are flaking?

arm64 / test (linux-arm64-integration-1-cpu)

Which tests are flaking?

TestLeaseGrantTimeToLiveExpired

Github Action / Prow Job link

https://github.com/etcd-io/etcd/actions/runs/8078398584/job/22070629565

Reason for failure (if possible)

    lease_test.go:146: 
        	Error Trace:	/home/runner/actions-runner/_work/etcd/etcd/tests/common/lease_test.go:146
        	            				/home/runner/actions-runner/_work/etcd/etcd/tests/framework/testutils/execute.go:38
        	            				/home/runner/actions-runner/_work/_tool/go/1.21.6/arm64/src/runtime/asm_arm64.s:1197
        	Error:      	Not equal: 
        	            	expected: -1
        	            	actual  : 1
        	Test:       	TestLeaseGrantTimeToLiveExpired/PeerAutoTLS

Anything else we need to know?

No response
@jmhbnz
Copy link
Member

jmhbnz commented Mar 5, 2024

Another instance https://github.com/etcd-io/etcd/actions/runs/8144423225/job/22282118668#step:5:1807.

@jmhbnz jmhbnz mentioned this issue Mar 5, 2024
Merged
@jmhbnz
Copy link
Member

jmhbnz commented Mar 31, 2024

Another instance https://github.com/etcd-io/etcd/actions/runs/8500313218/job/23282034904?pr=17677.

@fuweid
Copy link
Member

fuweid commented Apr 1, 2024

I think it's related to leader change. We can retry it 3 times if it's related to leader change.
/assign

24-03-31T19:20:48.1185991Z     logger.go:146: 2024-03-31T19:16:04.271Z	INFO	m2.raft	c81b22754673dbd7 became follower at term 4	{"member": "m2"}
2024-03-31T19:20:48.1188332Z     logger.go:146: 2024-03-31T19:16:04.271Z	INFO	m2.raft	raft.node: c81b22754673dbd7 changed leader from 3f9a3c374f2c7e67 to f9a0f85dc7e1942c at term 4	{"member": "m2"}
2024-03-31T19:20:48.1190852Z     logger.go:146: 2024-03-31T19:16:04.272Z	WARN	m0	Failed to check current member's leadership	{"member": "m0", "error": "etcdserver: leader changed"}
2024-03-31T19:20:48.1197275Z     logger.go:146: 2024-03-31T19:16:04.272Z	WARN	m0	Ignore the lease revoking request because current member isn't a leader	{"member": "m0", "local-member-id": 4583041779052084839}
2024-03-31T19:20:48.1201217Z     logger.go:146: 2024-03-31T19:16:04.572Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "c81b22754673dbd7", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "2.36516ms"}
2024-03-31T19:20:48.1205640Z     logger.go:146: 2024-03-31T19:16:04.572Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "3f9a3c374f2c7e67", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "2.52152ms"}
2024-03-31T19:20:48.1210353Z     logger.go:146: 2024-03-31T19:16:04.602Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "c81b22754673dbd7", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "685.36µs"}
2024-03-31T19:20:48.1215045Z     logger.go:146: 2024-03-31T19:16:04.603Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "3f9a3c374f2c7e67", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "811.24µs"}
2024-03-31T19:20:48.1217241Z     lease_test.go:146: 
2024-03-31T19:20:48.1218472Z         	Error Trace:	/home/runner/actions-runner/_work/etcd/etcd/tests/common/lease_test.go:146
2024-03-31T19:20:48.1220675Z         	            				/home/runner/actions-runner/_work/etcd/etcd/tests/framework/testutils/execute.go:38
2024-03-31T19:20:48.1222788Z         	            				/home/runner/actions-runner/_work/_tool/go/1.22.1/arm64/src/runtime/asm_arm64.s:1222
2024-03-31T19:20:48.1223781Z         	Error:      	Not equal: 
2024-03-31T19:20:48.1224504Z         	            	expected: -1
2024-03-31T19:20:48.1225333Z         	            	actual  : 1
2024-03-31T19:20:48.1226627Z         	Test:       	TestLeaseGrantTimeToLiveExpired/PeerAutoTLS
2024-03-31T19:20:48.1227698Z     cluster.go:1417: ========= Cluster termination started =====================

@fuweid fuweid mentioned this issue Apr 23, 2024
Merged
@ajaysundark ajaysundark mentioned this issue Jan 30, 2025
Closed
@ahrtr
Copy link
Member Author

ahrtr commented Feb 16, 2025

I think it's related to leader change. We can retry it 3 times if it's related to leader change.
/assign

Could you deliver a PR? thx

@fuweid
Copy link
Member

fuweid commented Feb 17, 2025

@ahrtr sorry for taking it so long. I was distracted. I will file pull request when it's ready

fuweid added a commit to fuweid/etcd that referenced this issue Feb 24, 2025
@fuweid
tests: deflakey TestLeaseGrantTimeToLiveExpired
ed0e40b 
The following change can reproduce the issue etcd-io#17504 by
`TESTCASE=TestLeaseGrantTimeToLiveExpired/PeerAutoTLS make test-e2e`. We
can retry it to reduce flakey possibility, if there is leader change.

```diff
diff --git a/tests/common/lease_test.go b/tests/common/lease_test.go
index afb27c330..dea8459dc 100644
--- a/tests/common/lease_test.go
+++ b/tests/common/lease_test.go
@@ -16,6 +16,7 @@ package common

 import (
        "context"
+       "sync"
        "testing"
        "time"

@@ -139,6 +140,18 @@ func TestLeaseGrantTimeToLiveExpired(t *testing.T) {
                                require.NoError(t, err)
                                require.Equal(t, int64(1), getResp.Count)

+                               var wg sync.WaitGroup
+                               defer wg.Wait()
+
+                               wg.Add(1)
+                               go func() {
+                                       defer wg.Done()
+                                       time.Sleep(1000 * time.Millisecond)
+                                       clus.(interface {
+                                               MoveLeader(ctx context.Context, t testing.TB, i int) error
+                                       }).MoveLeader(ctx, t, (clus.WaitLeader(t)+1)%len(clus.Members()))
+                               }()
+
                                time.Sleep(3 * time.Second)

                                ttlResp, err := cc.TimeToLive(ctx, leaseResp.ID, config.LeaseOption{})

```

Signed-off-by: Wei Fu <fuweid89@gmail.com>
@fuweid fuweid mentioned this issue Feb 24, 2025
Merged
fuweid added a commit to fuweid/etcd that referenced this issue Feb 24, 2025
@fuweid
tests: deflakey TestLeaseGrantTimeToLiveExpired
c81fb87 
The following change can reproduce the issue etcd-io#17504 by
`TESTCASE=TestLeaseGrantTimeToLiveExpired/PeerAutoTLS make test-e2e`. We
can retry it to reduce flakey possibility, if there is leader change.

```diff
diff --git a/tests/common/lease_test.go b/tests/common/lease_test.go
index afb27c330..dea8459dc 100644
--- a/tests/common/lease_test.go
+++ b/tests/common/lease_test.go
@@ -16,6 +16,7 @@ package common

 import (
        "context"
+       "sync"
        "testing"
        "time"

@@ -139,6 +140,18 @@ func TestLeaseGrantTimeToLiveExpired(t *testing.T) {
                                require.NoError(t, err)
                                require.Equal(t, int64(1), getResp.Count)

+                               var wg sync.WaitGroup
+                               defer wg.Wait()
+
+                               wg.Add(1)
+                               go func() {
+                                       defer wg.Done()
+                                       time.Sleep(1000 * time.Millisecond)
+                                       clus.(interface {
+                                               MoveLeader(ctx context.Context, t testing.TB, i int) error
+                                       }).MoveLeader(ctx, t, (clus.WaitLeader(t)+1)%len(clus.Members()))
+                               }()
+
                                time.Sleep(3 * time.Second)

                                ttlResp, err := cc.TimeToLive(ctx, leaseResp.ID, config.LeaseOption{})

```

Signed-off-by: Wei Fu <fuweid89@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fuweid fuweid

Labels
area/testing help wanted type/flake
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fuweid @ahrtr @jmhbnz