Wrong join results when joining a datetime column with a string column in some cases

[qw4990]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

create table t (
    a varchar(30),
    b datetime
);
insert into t values ("2000-01-01", "2000-01-01");
insert into t values ("2000xxxxxx", "2000-01-02");
select t1.a, t2.b from t t1, t t2 where t1.a != t2.b;

2. What did you expect to see?
   In MySQL:

mysql> select t1.a, t2.b from t t1, t t2 where t1.a != t2.b;
+------------+---------------------+
| a          | b                   |
+------------+---------------------+
| 2000xxxxxx | 2000-01-01 00:00:00 |
| 2000-01-01 | 2000-01-02 00:00:00 |
| 2000xxxxxx | 2000-01-02 00:00:00 |
+------------+---------------------+
3 rows in set, 2 warnings (0.00 sec)

3. What did you see instead?
   In TiDB:

mysql> select t1.a, t2.b from t t1, t t2 where t1.a != t2.b;
+------------+---------------------+
| a          | b                   |
+------------+---------------------+
| 2000-01-01 | 2000-01-02 00:00:00 |
+------------+---------------------+
1 row in set, 2 warnings (0.01 sec)

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

[ghost]

Confirming that this can still be reproduced against master:

drop table if exists t;
create table t (
    a varchar(30),
    b datetime
);
insert into t values ("2000-01-01", "2000-01-01");
insert into t values ("2000xxxxxx", "2000-01-02");
select t1.a, t2.b from t t1, t t2 where t1.a != t2.b;

..

mysql> select t1.a, t2.b from t t1, t t2 where t1.a != t2.b;
+------------+---------------------+
| a          | b                   |
+------------+---------------------+
| 2000-01-01 | 2000-01-02 00:00:00 |
+------------+---------------------+
1 row in set, 2 warnings (0.00 sec)

mysql> select tidb_version()\G
*************************** 1. row ***************************
tidb_version(): Release Version: v4.0.0-beta.2-782-gb72e47e6d
Edition: Community
Git Commit Hash: b72e47e6db8863c08e47714391cf937b0f1c3269
Git Branch: master
UTC Build Time: 2020-07-15 01:26:06
GoVersion: go1.13
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false
1 row in set (0.00 sec)

[qw4990]

This issue has the same root cause with #17868.

[qw4990]

I find this issue is hard to fix. The root cause is that our comparison operator is not null-aware.
For example, when processing 2000xxxxxx != 2000-01-01 00:00:00, the left operand 2000xxxxxx is converted to Datetime implicitly and since it's invalid, a Null is returned with a warning.
Then the comparison is converted to Null != 2000-01-01 00:00:00, since its result is Null, the row 2000xxxxxx, 2000-01-01 00:00:00 is removed from TiDB's results.

But MySQL also contradicts itself:

mysql> select * from t;
+------------+---------------------+
| a          | b                   |
+------------+---------------------+
| 2000-01-01 | 2000-01-01 00:00:00 |
| 2000xxxxxx | 2000-01-02 00:00:00 |
+------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from t where b != a;
+------------+---------------------+
| a          | b                   |
+------------+---------------------+
| 2000xxxxxx | 2000-01-02 00:00:00 |
+------------+---------------------+
1 row in set, 1 warning (0.00 sec)

mysql> select cast("2000xxxxxx" as datetime);
+--------------------------------+
| cast("2000xxxxxx" as datetime) |
+--------------------------------+
| NULL                           |
+--------------------------------+

mysql> select * from t where b != null;
Empty set (0.00 sec)

mysql> select * from t where b != "2000xxxxxx";
ERROR 1525 (HY000): Incorrect DATETIME value: '2000xxxxxx'

Since it's hard to fix and it only happens when comparing Datetime with invalid Datetime string, I suggest degrading this issue's severity and don't fix it now. @lzmhhh123 @SunRunAway

[lzmhhh123]

I agree with that. What's your opinion? @SunRunAway

[xzhangxian1008]

Is this bug fixed now?