expression: cast json_extract as bool #18948

Win-Man · 2020-08-03T09:36:33Z

What problem does this PR solve?

Issue Number: close #12233

Problem Summary:

What is changed and how it works?

Proposal: xxx

What's Changed:

How it Works:

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

CREATE TABLE `testjson` ( `id` int(11) NOT NULL AUTO_INCREMENT, `j` json DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB  DEFAULT CHARSET=utf8;

INSERT INTO testjson SET j='{"test":1}';

INSERT INTO testjson SET j='{"test":3}';

INSERT INTO testjson SET j='{"test":0}';

insert into testjson set j='{"test":"0"}';

insert into testjson set j='{"test":0.0}';

INSERT INTO testjson SET j='{"test":"a"}';

INSERT INTO testjson SET j='{"test":"aaabbb"}';

INSERT INTO testjson SET j='{"test":3.1415}';

INSERT INTO testjson SET j='{"test":-1}';

INSERT INTO testjson SET j='{"test":[]}';

INSERT INTO testjson SET j='{"test":[1,2]}';

INSERT INTO testjson SET j='{"test":["b","c"]}';

INSERT INTO testjson SET j='{"test":{"ke":"val"}}';

insert into testjson set j='{"test":"2015-07-27 09:43:47"}';

insert into testjson set j='{"test":"0000-00-00 00:00:00"}';

insert into testjson set j='{"test":"9778"}';

insert into testjson set j='{"test":"0778"}';

insert into testjson set j='{"test":"0000"}';
select * from testjson where json_extract(j,'$.test'); 

## mocktikv
mysql> select * from testjson where json_extract(j,'$.test');
+----+---------------------------------+
| id | j                               |
+----+---------------------------------+
|  1 | {"test": 1}                     |
|  2 | {"test": 3}                     |
|  4 | {"test": "0"}                   |
|  6 | {"test": "a"}                   |
|  7 | {"test": "aaabbb"}              |
|  8 | {"test": 3.1415}                |
|  9 | {"test": -1}                    |
| 10 | {"test": []}                    |
| 11 | {"test": [1, 2]}                |
| 12 | {"test": ["b", "c"]}            |
| 13 | {"test": {"ke": "val"}}         |
| 14 | {"test": "2015-07-27 09:43:47"} |
| 15 | {"test": "0000-00-00 00:00:00"} |
| 16 | {"test": "9778"}                |
| 17 | {"test": "0778"}                |
| 18 | {"test": "0000"}                |
+----+---------------------------------+
16 rows in set (0.01 sec)

mysql> desc select * from testjson where json_extract(j,'$.test');
+-------------------------+----------+-----------+----------------+---------------------------------------------+
| id                      | estRows  | task      | access object  | operator info                               |
+-------------------------+----------+-----------+----------------+---------------------------------------------+
| Selection_7             | 10000.00 | root      |                | json_extract(gangshen.testjson.j, "$.test") |
| └─TableReader_6         | 10000.00 | root      |                | data:TableFullScan_5                        |
|   └─TableFullScan_5     | 10000.00 | cop[tikv] | table:testjson | keep order:false, stats:pseudo              |
+-------------------------+----------+-----------+----------------+---------------------------------------------+
3 rows in set (0.01 sec)

insert into mysql.expr_pushdown_blacklist values('json_extract','tikv','');
admin reload expr_pushdown_blacklist;

mysql> select * from testjson where json_extract(j,'$.test');
+----+---------------------------------+
| id | j                               |
+----+---------------------------------+
|  1 | {"test": 1}                     |
|  2 | {"test": 3}                     |
|  4 | {"test": "0"}                   |
|  6 | {"test": "a"}                   |
|  7 | {"test": "aaabbb"}              |
|  8 | {"test": 3.1415}                |
|  9 | {"test": -1}                    |
| 10 | {"test": []}                    |
| 11 | {"test": [1, 2]}                |
| 12 | {"test": ["b", "c"]}            |
| 13 | {"test": {"ke": "val"}}         |
| 14 | {"test": "2015-07-27 09:43:47"} |
| 15 | {"test": "0000-00-00 00:00:00"} |
| 16 | {"test": "9778"}                |
| 17 | {"test": "0778"}                |
| 18 | {"test": "0000"}                |
+----+---------------------------------+
16 rows in set (0.00 sec)

mysql> desc select * from testjson where json_extract(j,'$.test');
+-------------------------+---------+-----------+----------------+---------------------------------------------+
| id                      | estRows | task      | access object  | operator info                               |
+-------------------------+---------+-----------+----------------+---------------------------------------------+
| TableReader_7           | 14.40   | root      |                | data:Selection_6                            |
| └─Selection_6           | 14.40   | cop[tikv] |                | json_extract(gangshen.testjson.j, "$.test") |
|   └─TableFullScan_5     | 18.00   | cop[tikv] | table:testjson | keep order:false, stats:pseudo              |
+-------------------------+---------+-----------+----------------+---------------------------------------------+
3 rows in set (0.00 sec)


## tikv
mysql>  select * from testjson where json_extract(j,'$.test');
+----+---------------------------------+
| id | j                               |
+----+---------------------------------+
|  1 | {"test": 1}                     |
|  2 | {"test": 3}                     |
|  4 | {"test": "0"}                   |
|  6 | {"test": "a"}                   |
|  7 | {"test": "aaabbb"}              |
|  8 | {"test": 3.1415}                |
|  9 | {"test": -1}                    |
| 10 | {"test": []}                    |
| 11 | {"test": [1, 2]}                |
| 12 | {"test": ["b", "c"]}            |
| 13 | {"test": {"ke": "val"}}         |
| 14 | {"test": "2015-07-27 09:43:47"} |
| 15 | {"test": "0000-00-00 00:00:00"} |
| 16 | {"test": "9778"}                |
| 17 | {"test": "0778"}                |
| 18 | {"test": "0000"}                |
+----+---------------------------------+
16 rows in set (0.07 sec)

mysql> desc  select * from testjson where json_extract(j,'$.test');
+-------------------------+---------+-----------+----------------+---------------------------------------------+
| id                      | estRows | task      | access object  | operator info                               |
+-------------------------+---------+-----------+----------------+---------------------------------------------+
| Selection_7             | 18.00   | root      |                | json_extract(gangshen.testjson.j, "$.test") |
| └─TableReader_6         | 18.00   | root      |                | data:TableFullScan_5                        |
|   └─TableFullScan_5     | 18.00   | cop[tikv] | table:testjson | keep order:false, stats:pseudo              |
+-------------------------+---------+-----------+----------------+---------------------------------------------+
3 rows in set (0.00 sec)

insert into mysql.expr_pushdown_blacklist values('json_extract','tikv','');
admin reload expr_pushdown_blacklist;

mysql>  select * from testjson where json_extract(j,'$.test');
Empty set (0.09 sec)

mysql> desc  select * from testjson where json_extract(j,'$.test');
+-------------------------+---------+-----------+----------------+---------------------------------------------+
| id                      | estRows | task      | access object  | operator info                               |
+-------------------------+---------+-----------+----------------+---------------------------------------------+
| TableReader_7           | 14.40   | root      |                | data:Selection_6                            |
| └─Selection_6           | 14.40   | cop[tikv] |                | json_extract(gangshen.testjson.j, "$.test") |
|   └─TableFullScan_5     | 18.00   | cop[tikv] | table:testjson | keep order:false, stats:pseudo              |
+-------------------------+---------+-----------+----------------+---------------------------------------------+
3 rows in set (0.00 sec)

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

CLAassistant · 2020-08-03T09:36:37Z

All committers have signed the CLA.

codecov · 2020-08-03T12:25:53Z

Codecov Report

Merging #18948 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #18948   +/-   ##
===========================================
  Coverage   79.4013%   79.4013%           
===========================================
  Files           546        546           
  Lines        148762     148762           
===========================================
  Hits         118119     118119           
  Misses        21173      21173           
  Partials       9470       9470

SunRunAway

You should add some test cases into code.

util/chunk/column.go

types/json/binary.go

…to json_extract_fix

Win-Man · 2020-08-04T15:42:04Z

@SunRunAway PTAL

SunRunAway

Would you like to add these following cases?

insert into testjson set j='""';
insert into testjson set j='null';
insert into testjson set j='0';

select * from testjson where j;

Win-Man · 2020-08-05T13:07:38Z

Would you like to add these following cases?

insert into testjson set j='""';
insert into testjson set j='null';
insert into testjson set j='0';

select * from testjson where j;

I've added some test cases.

SunRunAway

LGTM

lzmhhh123

LGTM

lzmhhh123 · 2020-08-07T08:42:49Z

/merge

ti-srebot · 2020-08-07T08:43:15Z

/run-all-tests

Win-Man added 2 commits August 1, 2020 15:36

fix mocktikv json expression

3ccb8d8

fix issue 12233

86995a6

Win-Man requested a review from a team as a code owner August 3, 2020 09:36

Win-Man requested review from SunRunAway and removed request for a team August 3, 2020 09:36

github-actions bot added sig/execution SIG execution component/expression labels Aug 3, 2020

Merge branch 'master' into json_extract_fix

c3d5e32

SunRunAway reviewed Aug 3, 2020

View reviewed changes

util/chunk/column.go Outdated Show resolved Hide resolved

types/json/binary.go Show resolved Hide resolved

Win-Man added 4 commits August 3, 2020 23:47

remove MysqlJSONS function

50ca3cf

Merge branch 'json_extract_fix' of https://github.com/Win-Man/tidb in…

78e9826

…to json_extract_fix

add test case and fix literal to bool

4008244

go fmt

eb5d59b

SunRunAway reviewed Aug 5, 2020

View reviewed changes

add more test case

791a811

SunRunAway reviewed Aug 7, 2020

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 7, 2020

SunRunAway requested a review from lzmhhh123 August 7, 2020 03:34

Win-Man added 2 commits August 7, 2020 14:11

update test case

05a4b97

Merge branch 'master' into json_extract_fix

b6488e4

lzmhhh123 reviewed Aug 7, 2020

View reviewed changes

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Aug 7, 2020

ti-srebot approved these changes Aug 7, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Aug 7, 2020

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 7, 2020

ti-srebot merged commit 2b05ffd into pingcap:master Aug 7, 2020

wshwsh12 mentioned this pull request Dec 16, 2020

copr: implement json cast as bool tikv/tikv#9286

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

expression: cast json_extract as bool #18948

expression: cast json_extract as bool #18948

Win-Man commented Aug 3, 2020 •

edited

Loading

CLAassistant commented Aug 3, 2020 •

edited

Loading

codecov bot commented Aug 3, 2020 •

edited

Loading

SunRunAway left a comment

Win-Man commented Aug 4, 2020

SunRunAway left a comment •

edited

Loading

Win-Man commented Aug 5, 2020

SunRunAway left a comment

lzmhhh123 left a comment

lzmhhh123 commented Aug 7, 2020

ti-srebot commented Aug 7, 2020

expression: cast json_extract as bool #18948

expression: cast json_extract as bool #18948

Conversation

Win-Man commented Aug 3, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

CLAassistant commented Aug 3, 2020 • edited Loading

codecov bot commented Aug 3, 2020 • edited Loading

Codecov Report

SunRunAway left a comment

Choose a reason for hiding this comment

Win-Man commented Aug 4, 2020

SunRunAway left a comment • edited Loading

Choose a reason for hiding this comment

Win-Man commented Aug 5, 2020

SunRunAway left a comment

Choose a reason for hiding this comment

lzmhhh123 left a comment

Choose a reason for hiding this comment

lzmhhh123 commented Aug 7, 2020

ti-srebot commented Aug 7, 2020

Win-Man commented Aug 3, 2020 •

edited

Loading

CLAassistant commented Aug 3, 2020 •

edited

Loading

codecov bot commented Aug 3, 2020 •

edited

Loading

SunRunAway left a comment •

edited

Loading