The variable tidb_allow_fallback_to_tikv can't take effect when the tiflash nodes stop

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tpcc 10k warehouses and tiflash replica 2
tiup cluster stop xxx -R tiflash
select sum(s_order_cnt) from stock where s_i_id=1; -- ERROR 9012 (HY000): TiFlash server timeout
set @@tidb_allow_fallback_to_tikv="tiflash";
select sum(s_order_cnt) from stock where s_i_id=1; -- ERROR 9005 (HY000): Region is unavailable

The plan is

+--------------------------------+---------------+-------------------+---------------+---------------------------------------------------------------+
| id                             | estRows       | task              | access object | operator info                                                 |
+--------------------------------+---------------+-------------------+---------------+---------------------------------------------------------------+
| StreamAgg_27                   | 1.00          | root              |               | funcs:sum(Column#21)->Column#19                               |
| └─TableReader_28               | 1.00          | root              |               | data:StreamAgg_9                                              |
|   └─StreamAgg_9                | 1.00          | batchCop[tiflash] |               | funcs:sum(Column#22)->Column#21                               |
|     └─Projection_35            | 10064.41      | batchCop[tiflash] |               | cast(tpcc.stock.s_order_cnt, decimal(32,0) BINARY)->Column#22 |
|       └─Selection_26           | 10064.41      | batchCop[tiflash] |               | eq(tpcc.stock.s_i_id, 1)                                      |
|         └─TableFullScan_25     | 1000000000.00 | batchCop[tiflash] | table:stock   | keep order:false                                              |
+--------------------------------+---------------+-------------------+---------------+---------------------------------------------------------------+
6 rows in set (0.05 sec)

2. What did you expect to see? (Required)

The last query fallback to tikv

3. What did you see instead (Required)

ERROR 9005 (HY000): Region is unavailable

4. What is your TiDB version? (Required)

Release Version: v4.0.0-beta.2-2791-g207ce344c-dirty
Edition: Community
Git Commit Hash: 207ce34
Git Branch: master
UTC Build Time: 2021-05-02 15:24:42
GoVersion: go1.13
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[winoros]

Is the environment available? Or a more detailed log stack?
@lilinghai

[lilinghai]

The second query log is

[2021/05/06 12:12:50.161 +00:00] [WARN] [backoff.go:332] ["regionMiss backoffer.maxSleep 10000ms is exceeded, errors:\nCannot find region with TiFlash peer at 2021-05-06T12:12:49.038191558Z\nCannot find region with TiFlash peer at 2021-05-06T12:12:49.601745515Z\nCannot find region with TiFlash peer at 2021-05-06T12:12:50.161559434Z"]
[2021/05/06 12:12:50.162 +00:00] [INFO] [conn.go:878] ["command dispatched failed"] [conn=57] [connInfo="id:57, addr:100.64.10.10:56638 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="select count(*) from stock"] [txn_mode=PESSIMISTIC] [err="[tikv:9005]Region is unavailable\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/agent/workspace/tidb_master/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/home/jenkins/agent/workspace/tidb_master/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/copr.buildBatchCopTasks\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:155\ngithub.com/pingcap/tidb/store/copr.(*CopClient).sendBatch\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:181\ngithub.com/pingcap/tidb/store/copr.(*CopClient).Send\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:73\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/distsql/distsql.go:77\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/distsql/distsql.go:126\ngithub.com/pingcap/tidb/executor.selectResultHook.SelectResult\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/table_reader.go:51\ngithub.com/pingcap/tidb/executor.(*TableReaderExecutor).buildResp\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/table_reader.go:241\ngithub.com/pingcap/tidb/executor.(*TableReaderExecutor).Open\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/table_reader.go:157\ngithub.com/pingcap/tidb/executor.(*baseExecutor).Open\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/executor.go:163\ngithub.com/pingcap/tidb/executor.(*StreamAggExec).Open\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/aggregate.go:1104\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/executor/adapter.go:336\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/session/session.go:1576\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/session/session.go:1471\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/driver_tidb.go:218\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:1697\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:1569\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:1103\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/conn.go:861\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/tidb_master/go/src/github.com/pingcap/tidb/server/server.go:481\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

I will offer the env to you with other chat tools.

[ichn-hu]

/unlabel sig/execution

[ichn-hu]

/label sig/planner

[ti-chi-bot]

@ichn-hu: The label(s) sig/planner cannot be applied. These labels are supported: CHANGELOG, challenge-program, compatibility-breaker, contribution, correctness, first-time-contributor, good-first-issue, hacktoberfest, high-performance, hptc, need-more-info, needs-cherry-pick-3.0, needs-cherry-pick-3.1, needs-cherry-pick-4.0, needs-cherry-pick-5.0-rc, needs-cherry-pick-5.0, proposal, release-note, require-LGT3, security, tools.

In response to this:

/label sig/planner

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions