Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests: global timestamp fallback in integrations/tso #6634

Closed
lhy1024 opened this issue Jun 19, 2023 · 3 comments · Fixed by #6655 or #6740
Closed

tests: global timestamp fallback in integrations/tso #6634

lhy1024 opened this issue Jun 19, 2023 · 3 comments · Fixed by #6655 or #6740
Labels
type/ci The issue is related to CI.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jun 19, 2023 •
edited
Loading

Flaky Test

Which jobs are failing

/home/runner/work/pd/pd/tests/integrations/tso

2023-06-19T08:41:19.6750574Z panic: global timestamp fallback, new ts (1687164079143, 1) <= the last one (1687164080662, 2861). keyspace: 4294967295, keyspace group: 0
2023-06-19T08:41:19.6750591Z 
2023-06-19T08:41:19.6750703Z goroutine 970548 [running]:
2023-06-19T08:41:19.6750945Z github.com/tikv/pd/client.(*tsoClient).compareAndSwapTS(0xc007255e00, {0x4236587, 0x6}, 0x188d2d0cc27, 0x1, 0x0, 0x1)
2023-06-19T08:41:19.6751144Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:742 +0x536
2023-06-19T08:41:19.6751408Z github.com/tikv/pd/client.(*tsoClient).processRequests(0xc007255e00, {0x516f020, 0xc0021d1aa0}, {0x4236587, 0x6}, 0xc006c50820, {0x0, 0x0, 0x0})
2023-06-19T08:41:19.6751602Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:720 +0x4f0
2023-06-19T08:41:19.6751857Z github.com/tikv/pd/client.(*tsoClient).handleDispatcher(0xc007255e00, {0x5189b70?, 0xc006c507d0}, {0x4236587, 0x6}, 0xc006c50820)
2023-06-19T08:41:19.6752051Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:445 +0x1b25
2023-06-19T08:41:19.6752245Z created by github.com/tikv/pd/client.(*tsoClient).createTSODispatcher
2023-06-19T08:41:19.6752433Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:290 +0x4f7
2023-06-19T08:41:19.6752605Z FAIL	github.com/tikv/pd/tests/integrations/tso	237.122s
2023-06-19T08:41:19.6752683Z FAIL
2023-06-19T08:41:19.6752977Z make[1]: Leaving directory '/home/runner/work/pd/pd/tests/integrations/tso'

CI link

https://github.com/tikv/pd/actions/runs/5309547422/jobs/9610545886
https://github.com/tikv/pd/actions/runs/5318919915/jobs/9631236217?pr=6626

Reason for failure (if possible)

Anything else
@lhy1024 lhy1024 added the type/ci The issue is related to CI. label Jun 19, 2023
@JmPotato JmPotato mentioned this issue Jun 21, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 21, 2023
@JmPotato
tests: fix TestTSOKeyspaceGroupSplitClient to avoid unexpected panic (#…
3431779 
…6655)

close #6634

Fix `TestTSOKeyspaceGroupSplitClient` to avoid unexpected panic

Signed-off-by: JmPotato <ghzpotato@gmail.com>
@JmPotato
Copy link
Member

Encounter again: https://github.com/tikv/pd/actions/runs/5367692418/jobs/9738042834?pr=6625

@JmPotato JmPotato reopened this Jun 25, 2023
@lhy1024
Copy link
Contributor Author

lhy1024 commented Jun 28, 2023

Encounter again: https://github.com/tikv/pd/actions/runs/5367692418/jobs/9738042834?pr=6625

Can it be closed?

@JmPotato
Copy link
Member

JmPotato commented Jul 3, 2023

Encounter again: tikv/pd/actions/runs/5367692418/jobs/9738042834?pr=6625

Can it be closed?

Even if it's not a bug, it's still a flasky test. Let's keep this issue open until it's stable.

@JmPotato JmPotato mentioned this issue Jul 3, 2023
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in #6740 Jul 6, 2023
ti-chi-bot bot added a commit that referenced this issue Jul 6, 2023
@JmPotato @ti-chi-bot
client, tests: allow TSO fallback happens in TestMixedTSODeployment (#…
b176a0a 
…6740)

close #6634

Introduce `WithAllowTSOFallback` client option to bypass the panic in `TestMixedTSODeployment`.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @rleungx
tests: fix TestTSOKeyspaceGroupSplitClient to avoid unexpected panic (t…
8efcdf4 
…ikv#6655)

close tikv#6634

Fix `TestTSOKeyspaceGroupSplitClient` to avoid unexpected panic

Signed-off-by: JmPotato <ghzpotato@gmail.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
client, tests: allow TSO fallback happens in TestMixedTSODeployment (t…
80dfa01 
…ikv#6740)

close tikv#6634

Introduce `WithAllowTSOFallback` client option to bypass the panic in `TestMixedTSODeployment`.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @rleungx
tests: fix TestTSOKeyspaceGroupSplitClient to avoid unexpected panic (t…
e73c62e 
…ikv#6655)

close tikv#6634

Fix `TestTSOKeyspaceGroupSplitClient` to avoid unexpected panic

Signed-off-by: JmPotato <ghzpotato@gmail.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@JmPotato @ti-chi-bot @rleungx
client, tests: allow TSO fallback happens in TestMixedTSODeployment (t…
72674b8 
…ikv#6740)

close tikv#6634

Introduce `WithAllowTSOFallback` client option to bypass the panic in `TestMixedTSODeployment`.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
2 participants
@JmPotato @lhy1024