Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In some cases, the error code of 1105 does not give specific error information #26147

Closed
vivid392845427 opened this issue Jul 12, 2021 · 7 comments · Fixed by #26647
Closed
Assignees
Labels
severity/moderate sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@vivid392845427
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

run tipocket bank2,set nemesis type:pod-failure, the testcase fail, error message is null; for example:

2021/07/11 12:03:06 control.go:478: [info] recover nemesis type:pod-failure,duration:1m34s,node:node[comp=pd,ip=default-1626001720-pd-2.default-1626001720-pd-peer.tipocket-bank2.svc:2379,ns=tipocket-bank2,pod=default-1626001720-pd-2],invoke_args:[],recover_args:[]...

2021/07/11 12:03:06 kill.go:155: [info] unapply nemesis pod-failure on node default-1626001720-pd-2(ns:tipocket-bank2)

2021/07/11 12:03:06 client.go:397: [fatal] [bank2] ADMIN CHECK TABLE bank2_accounts fails: Error 1105:

2. What did you expect to see? (Required)

return error code and detail error message

3. What did you see instead (Required)

Error 1105:

4. What is your TiDB version? (Required)

Release Version: v5.1.0-4-ge96c2a6da
Edition: Community
Git Commit Hash: e96c2a6
Git Branch: release-5.1
UTC Build Time: 2021-07-07 12:16:42
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

@vivid392845427 vivid392845427 added the type/bug The issue is confirmed as a bug. label Jul 12, 2021
@wjhuang2016 wjhuang2016 self-assigned this Jul 16, 2021
@wjhuang2016
Copy link
Member

BoPDRPC      = NewConfig("pdRPC", &metrics.BackoffHistogramPD, NewBackoffFnCfg(500, 3000, EqualJitter), tikverr.NewErrPDServerTimeout(""))

No error message for PD server timeout.

@wjhuang2016
Copy link
Member

/cc @disksing PTAL

@wjhuang2016 wjhuang2016 assigned disksing and unassigned wjhuang2016 Jul 27, 2021
@disksing
Copy link
Contributor

Thanks we wil take a look. /cc @AndreMouche

@AndreMouche
Copy link
Contributor

BoPDRPC      = NewConfig("pdRPC", &metrics.BackoffHistogramPD, NewBackoffFnCfg(500, 3000, EqualJitter), tikverr.NewErrPDServerTimeout(""))

No error message for PD server timeout.

I think it is not caused by PD server timeout. The error for PD server timeout in tikv-client should be converted to a normal PD error for tidb in driver and it have its special error code 9001

By the way, @vivid392845427 could you provide the log of tidb/pd/tikv?

@vivid392845427
Copy link
Author

tidb log:
[2021/07/27 02:53:34.510 +00:00] [INFO] [conn.go:995] ["command dispatched failed"] [conn=411] [connInfo="id:411, addr:172.18.180.211:54194 status:11, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:1, autocommit:1"] [sql="ADMIN CHECK TABLE bank2_accounts"] [txn_mode=PESSIMISTIC] [err="\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/juju_adaptor.go:15\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionCache).loadRegion\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210723073051-3d747cd203ba/internal/locate/region_cache.go:1271\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionCache).findRegionByKey\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210723073051-3d747cd203ba/internal/locate/region_cache.go:755\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionCache).LocateKey\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210723073051-3d747cd203ba/internal/locate/region_cache.go:726\ngithub.com/pingcap/tidb/store/copr.(*RegionCache).SplitKeyRangesByLocations\n\t/home/jenkins/agent/workspace/build-tidb/tidb/store/copr/region_cache.go:65\ngithub.com/pingcap/tidb/store/copr.buildCopTasks\n\t/home/jenkins/agent/workspace/build-tidb/tidb/store/copr/coprocessor.go:162\ngithub.com/pingcap/tidb/store/copr.(*CopClient).Send\n\t/home/jenkins/agent/workspace/build-tidb/tidb/store/copr/coprocessor.go:81\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/build-tidb/tidb/distsql/distsql.go:77\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/build-tidb/tidb/distsql/distsql.go:126\ngithub.com/pingcap/tidb/executor.(*IndexLookUpExecutor).startIndexWorker.func1\n\t/home/jenkins/agent/workspace/build-tidb/tidb/executor/distsql.go:578\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2021/07/27 02:53:35.501 +00:00] [WARN] [grpclogger.go:89] ["grpc: addrConn.createTransport failed to connect to {debug1-pd-2.debug1-pd-peer.tipocket-bank2.svc:2379 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 172.18.172.215:2379: connect: connection refused". Reconnecting..."] [system=grpc] [grpc_log=true]

@AndreMouche
Copy link
Contributor

According to the log of tidb, it is a bug caused by that the driver does not convert the tikv.err to tidb.err in function tidb/store/copr/region_cache.go:65

@ti-srebot
Copy link
Contributor

Please edit this comment or add a new comment to complete the following information

Not a bug

  1. Remove the 'type/bug' label
  2. Add notes to indicate why it is not a bug

Duplicate bug

  1. Add the 'type/duplicate' label
  2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
severity/moderate sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants