executor: unexpected result for hash join on enum and set columns

[bb7133]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb> create table t(a int, b enum('A', 'B'));
Query OK, 0 rows affected (0.01 sec)

tidb> create table t1(a1 int, b1 enum('B', 'A'));
Query OK, 0 rows affected (0.01 sec)

tidb> insert into t values (1, 'A');
Query OK, 1 row affected (0.01 sec)

tidb> insert into t1 values (1, 'A');
Query OK, 1 row affected (0.00 sec)

tidb> select /*+ HASH_JOIN(t, t1) */ * from t join t1 on t.b = t1.b1;

Step 2 and 3 followed below behaves exact the same if enum is replaced with set for the step above.

2. What did you expect to see? (Required)

mysql> select * from t join t1 on t.b = t1.b1;
+------+------+------+------+
| a    | b    | a1   | b1   |
+------+------+------+------+
|    1 | A    |    1 | A    |
+------+------+------+------+
1 row in set (0.00 sec)

3. What did you see instead (Required)

tidb> select /*+ HASH_JOIN(t, t1) */ * from t join t1 on t.b = t1.b1;
Empty set (0.01 sec)

4. Affected version (Required)

v4.0.x

5. Root Cause Analysis

It looks that when building the HashTable used by HashJoin, the hash key is built from the offset of enum/set value:

https://github.com/pingcap/tidb/blob/master/util/codec/codec.go#L571

When the values of enum/set have different offsets by their definitions, the join result would be wrong.