regression issue: return empty result

[ChenPeng2013]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

use test;
drop table if exists PK_HP9231;
CREATE TABLE `PK_HP9231` (
  `COL1` mediumint(45) unsigned NOT NULL DEFAULT '25' COMMENT 'NUMERIC PK',
  `COL2` varchar(20) DEFAULT NULL,
  `COL4` datetime DEFAULT NULL,
  `COL3` bigint(20) NOT NULL,
  `COL5` float DEFAULT NULL,
  PRIMARY KEY (`COL1`,`COL3`) /*T![clustered_index] NONCLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
PARTITION BY HASH( `COL1`*`COL3` )
PARTITIONS 13;
insert into PK_HP9231(col1, col3) values(0, 3522101843073676459);
SELECT col1, COL3 FROM PK_HP9231 WHERE COL1 IN (0,14158354938390,0) AND COL3 IN (3522101843073676459,-2846203247576845955,838395691793635638);

2. What did you expect to see? (Required)

mysql> SELECT col1, COL3 FROM PK_HP9231 WHERE COL1 IN (0,14158354938390,0) AND COL3 IN (3522101843073676459,-2846203247576845955,838395691793635638);
+------+---------------------+
| col1 | COL3                |
+------+---------------------+
|    0 | 3522101843073676459 |
+------+---------------------+
1 row in set (0.00 sec)

3. What did you see instead (Required)

This issue is introduced by #26894

release-5.1 & master

mysql> SELECT col1, COL3 FROM PK_HP9231 WHERE COL1 IN (0,14158354938390,0) AND COL3 IN (3522101843073676459,-2846203247576845955,838395691793635638);
Empty set, 1 warning (0.00 sec)

mysql> show warnings;
+---------+------+----------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                                |
+---------+------+----------------------------------------------------------------------------------------+
| Warning | 1690 | BIGINT UNSIGNED value is out of range in '(test.pk_hp9231.col1 * test.pk_hp9231.col3)' |
+---------+------+----------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

4. What is your TiDB version? (Required)

Release Version: v5.1.1-58-gdc079337e
Edition: Community
Git Commit Hash: dc079337ef0c9d78eec2d2acb1eb31e12117d29b
Git Branch: release-5.1
UTC Build Time: 2021-09-22 02:01:55
GoVersion: go1.16.5
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.