hashAgg panic when UpdatePartialResult

[wjhuang2016]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

CREATE TABLE `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262` (
  `b47ed459-807c-4e80-987f-87b96248ff05` float NOT NULL DEFAULT '1.4563651',
  `681fd446-9ea1-4017-ab9f-590d9d4544e7` tinyint(4) NOT NULL DEFAULT '-21',
  PRIMARY KEY (`b47ed459-807c-4e80-987f-87b96248ff05`,`681fd446-9ea1-4017-ab9f-590d9d4544e7`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin;
INSERT INTO `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262` VALUES (6162.3276,0),(8178.5693,-27),(78.69162,-128),(5093.862,49),(2537.274,17),(9563.064,-44),(1878.5768,41);

select  /*+ use_index_merge( `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262` ) */   bit_length( `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262`.`b47ed459-807c-4e80-987f-87b96248ff05` ) as r0 , avg( distinct  `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262`.`681fd446-9ea1-4017-ab9f-590d9d4544e7` ) as r1 from `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262` where `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262`.`b47ed459-807c-4e80-987f-87b96248ff05` > 1422.3551427171312 or IsNull( `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262`.`681fd446-9ea1-4017-ab9f-590d9d4544e7` ) group by `5aa6e0cc-1fa2-4625-b8ff-74c648ce3262`.`b47ed459-807c-4e80-987f-87b96248ff05`;

2. What did you expect to see? (Required)

No error

3. What did you see instead (Required)

[err="runtime error: index out of range [0] with length 0\ngithub.com/pingcap/tidb/executor.recoveryProjection\n\t/Users/bba/pc/tidb/executor/projection.go:460\ngithub.com/pingcap/tidb/executor.(*projectionInputFetcher).run.func1\n\t/Users/bba/pc/tidb/executor/projection.go:360\nruntime.gopanic\n\t/Users/bba/.gvm/gos/go1.19/src/runtime/panic.go:884\nruntime.goPanicIndex\n\t/Users/bba/.gvm/gos/go1.19/src/runtime/panic.go:113\ngithub.com/pingcap/tidb/util/chunk.(*Column).IsNull\n\t/Users/bba/pc/tidb/util/chunk/column.go:169\ngithub.com/pingcap/tidb/util/chunk.Row.IsNull\n\t/Users/bba/pc/tidb/util/chunk/row.go:218\ngithub.com/pingcap/tidb/expression.(*Column).EvalReal\n\t/Users/bba/pc/tidb/expression/column.go:428\ngithub.com/pingcap/tidb/executor/aggfuncs.(*firstRow4Float32).UpdatePartialResult\n\t/Users/bba/pc/tidb/executor/aggfuncs/func_first_row.go:185\ngithub.com/pingcap/tidb/executor.(*HashAggExec).execute\n\t/Users/bba/pc/tidb/executor/aggregate.go:1044\ngithub.com/pingcap/tidb/executor.(*HashAggExec).unparallelExec\n\t/Users/bba/pc/tidb/executor/aggregate.go:966\ngithub.com/pingcap/tidb/executor.(*HashAggExec).Next\n\t/Users/bba/pc/tidb/executor/aggregate.go:785\ngithub.com/pingcap/tidb/executor.Next\n\t/Users/bba/pc/tidb/executor/executor.go:328\ngithub.com/pingcap/tidb/executor.(*projectionInputFetcher).run\n\t/Users/bba/pc/tidb/executor/projection.go:384\nruntime.goexit\n\t/Users/bba/.gvm/gos/go1.19/src/runtime/asm_arm64.s:1165"]

If remove the index merge hint, we got:

ERROR 1105 (HY000): should ensure all columns have the same length, expect 2, but got 0

4. What is your TiDB version? (Required)

master

[ywqzzy]

Can't reproduce it, But it is really a problem that can happen.

[ywqzzy]

Can't reproduce it, But it is really a problem that can happen.

Maybe it is related to chunks that come from tikv were broken. But I can't find any evidence.

[wjhuang2016]

CREATE TABLE `51f733a0-38f8-43c0-9d8f-65cb595de6cb` (
  `0c847db1-c6fe-4a0a-b94b-7b1db90baecf` set('2','b3y6i','m','tl4y','rr76e','tto','c39i','k4','0','s','t9') NOT NULL DEFAULT 'tl4y,k4',
  `3faa4ffc-299d-45c8-ad29-6193149a2a67` int(11) NOT NULL DEFAULT '-1622310057',
  PRIMARY KEY (`0c847db1-c6fe-4a0a-b94b-7b1db90baecf`,`3faa4ffc-299d-45c8-ad29-6193149a2a67`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=gbk COLLATE=gbk_chinese_ci COMMENT='a5e6239a-44f1-472e-8fa2-7be46d9bff47'
PARTITION BY HASH (`3faa4ffc-299d-45c8-ad29-6193149a2a67`) PARTITIONS 10;

INSERT INTO `51f733a0-38f8-43c0-9d8f-65cb595de6cb` VALUES ('tto',-1369524874),('s',609852990),('tl4y',-1533124452),('tto',-912893072),('0',1695262582),('tto',1190930211),('b3y6i',-2054779085),('b3y6i',-1744747545),('tto',-882858939);
/*!40000 ALTER TABLE `51f733a0-38f8-43c0-9d8f-65cb595de6cb` ENABLE KEYS */;

select     `51f733a0-38f8-43c0-9d8f-65cb595de6cb`.`0c847db1-c6fe-4a0a-b94b-7b1db90baecf` as r0 , avg( distinct  `51f733a0-38f8-43c0-9d8f-65cb595de6cb`.`3faa4ffc-299d-45c8-ad29-6193149a2a67` ) as r1 from `51f733a0-38f8-43c0-9d8f-65cb595de6cb`  group by `51f733a0-38f8-43c0-9d8f-65cb595de6cb`.`0c847db1-c6fe-4a0a-b94b-7b1db90baecf`  order by r0,r1 ;

[ywqzzy]

CREATE TABLE t (
  c1 set('2','b3y6i','m','tl4y','rr76e','tto','c39i','k4','0','s','t9'),
  c2 int,
  PRIMARY KEY (c1, c2)
)
PARTITION BY HASH (c2) PARTITIONS 2;

INSERT INTO t VALUES ('tto',-1369524874),('s',609852990),('tl4y',-1533124452),('tto',-912893072),('0',1695262582),('tto',1190930211),('b3y6i',-2054779085),('b3y6i',-1744747545);

select  t.c1 as r0 , avg( distinct  t.c2 ) as r1 from t  group by t.c1;

Minimal sql to reproduce the bug.

[ywqzzy]

In logicalPlan rewrite phase, tidb will try to push down aggregation.
In this case, the aggregationPushDownSolver will call tryAggPushDownForUnion to push down agg under union.
We can see that Projection_16 is the pushed down plan under PartitionUnion_15.
Here is the implement of tryAggPushDownForUnion

In line 437, we split agg into partial Agg and final Agg.
After the split, we can see the final agg's AggFuncs[1].Args[0].UniqueID = 4, while this column is ref to column c1, which have UniqueID = 1.

Then in the postOptimize phase, InjectExtraProjection add Projection_22.
Column#4 and c1 ref to the same location in memory, when the projection call evaluatorSuite, the columnEvaluator will swap the column with input. It will first swap column#4, then swap c1. Since column#4 will be set to one empty chunk, when we swap column c1, the result will be empty too. Then the bug ocurred.

[ywqzzy]

Same with #40184 #40315

[kennedy8312]

Seeing the issue on latest nightly as well.

[kennedy8312]

/component tablepartition

[winoros]

Whatever, the 8.1 can get correct result with dynamic pruning is really really set by default whatever stats exists or not.

[winoros]

We keep it open unless the static pruning is totally removed, but tidb starting from 8.1 will not panic by default.

Starting from v8.1, you need to set @@tidb_skip_missing_partition_stats=off to reproduce the issue.