Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestGetMinTS is flaky #6453

Closed
rleungx opened this issue May 12, 2023 · 0 comments · Fixed by #6457
Closed

TestGetMinTS is flaky #6453

rleungx opened this issue May 12, 2023 · 0 comments · Fixed by #6457
Assignees
@binshi-bing
Labels
type/ci The issue is related to CI.

Comments

@rleungx
Copy link
Member

rleungx commented May 12, 2023

Flaky Test

Which jobs are failing

--- FAIL: TestMicroserviceTSOClient (70.53s)
    --- FAIL: TestMicroserviceTSOClient/TestGetMinTS (7.89s)
        client_test.go:267: 
            	Error Trace:	/home/runner/work/pd/pd/tests/integrations/tso/client_test.go:267
            	            				/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/asm_amd64.s:1598
            	Error:      	Received unexpected error:
            	            	rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster
            	            	github.com/tikv/pd/client.(*tsoTSOStream).processRequests
            	            		/home/runner/work/pd/pd/client/tso_stream.go:190
            	            	github.com/tikv/pd/client.(*tsoClient).processRequests
            	            		/home/runner/work/pd/pd/client/tso_dispatcher.go:711
            	            	github.com/tikv/pd/client.(*tsoClient).handleDispatcher
            	            		/home/runner/work/pd/pd/client/tso_dispatcher.go:445
            	            	runtime.goexit
            	            		/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/asm_amd64.s:1598
            	            	github.com/tikv/pd/client.(*tsoRequest).Wait
            	            		/home/runner/work/pd/pd/client/tso_dispatcher.go:94
            	            	github.com/tikv/pd/client.(*client).GetTS
            	            		/home/runner/work/pd/pd/client/client.go:717
            	            	github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestGetMinTS.func1
            	            		/home/runner/work/pd/pd/tests/integrations/tso/client_test.go:266
            	            	runtime.goexit
            	            		/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/asm_amd64.s:1598
            	Test:       	TestMicroserviceTSOClient/TestGetMinTS
        client_test.go:269: 
            	Error Trace:	/home/runner/work/pd/pd/tests/integrations/tso/client_test.go:269
            	            				/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/asm_amd64.s:1598
            	Error:      	"441418138665353238" is not less than "0"
            	Test:       	TestMicroserviceTSOClient/TestGetMinTS

CI link

https://github.com/tikv/pd/actions/runs/4956285469/jobs/8866547918?pr=6409

Reason for failure (if possible)

Anything else
@rleungx rleungx added the type/ci The issue is related to CI. label May 12, 2023
@binshi-bing binshi-bing mentioned this issue May 12, 2023
Merged
binshi-bing added a commit to binshi-bing/pd that referenced this issue May 12, 2023
@binshi-bing
Disable TestGetMinTS due to tikv#6453
7f3b456 
Signed-off-by: Bin Shi <binshi.bing@gmail.com>
@binshi-bing binshi-bing mentioned this issue May 12, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 12, 2023
@binshi-bing
Disable TestGetMinTS since it's not stable (#6459)
511115f 
ref #6453

Disable TestGetMinTS due to #6453

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
ti-chi-bot bot pushed a commit that referenced this issue May 16, 2023
@binshi-bing
Fix "non-default keyspace groups use the same timestamp path by mista…
39e58df 
…ke" (#6457)

close #6453, close #6465

The tso servers are loading keyspace groups asynchronously. Make sure all keyspace groups
are available for serving tso requests from corresponding keyspaces by querying
IsKeyspaceServing(keyspaceID, the Desired KeyspaceGroupID). if use default keyspace group id
in the query, it will always return true as the keyspace will be served by default keyspace group
before the keyspace groups are loaded.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Disable TestGetMinTS since it's not stable (tikv#6459)
09c80ba 
ref tikv#6453

Disable TestGetMinTS due to tikv#6453

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @rleungx
Fix "non-default keyspace groups use the same timestamp path by mista…
20571e4 
…ke" (tikv#6457)

close tikv#6453, close tikv#6465

The tso servers are loading keyspace groups asynchronously. Make sure all keyspace groups
are available for serving tso requests from corresponding keyspaces by querying
IsKeyspaceServing(keyspaceID, the Desired KeyspaceGroupID). if use default keyspace group id
in the query, it will always return true as the keyspace will be served by default keyspace group
before the keyspace groups are loaded.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@binshi-bing binshi-bing

Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix "non-default keyspace groups use the same timestamp path by mistake" binshi-bing/pd
2 participants
@rleungx @binshi-bing