Call For Participation: SIG-Exec 2020/Q1 Plan

[SunRunAway]

All issues are open and welcoming to contributors. You can join #sig-exec on tidb community slack to discuss and get help from someone.

Objective 1: Improve the performance of the execution engine

Improve the performance of TiDB’s execution engine by optimizing some executors.

KR 1: Improve the performance of the Window, Aggregate, Join, Sort executors

issue	difficulty	assignee	PRs
Improve the performance of WindowExec by using sliding window or segment tree #12967 Optimize the Window executor by using sliding windows	medium	@mmyj, @SunRunAway	N/A
PCP-4: Improve the performance of WindowExec by using multi-thread hash grouping #12966 Optimize the Window executor by using multi-thread hash grouping	medium	@pingyu, @SunRunAway	#14238
PCP: Vectorize the merge join executor #14216 Vectorized Merge Join	medium	@catror, @qw4990	#14458
design vectorized aggregation functions #14217 Design vectorized aggregation functions	hard	help wanted	N/A
Investigate and propose parallel sort-merge join #14441 Implement Parallel Sort-Merge Join	hard	help wanted	N/A
Refine the implementation of HashJoinExecutor #14466 Refine the implementation of HashJoinExecution	medium	help wanted	N/A
#TODO Investigate Parallel Hash Join	medium	help wanted	N/A
Improve the performance of Sort by parallel #14417 Implement Parallel Sort	medium	@wshwsh12	N/A

KR 2: Reduce the memory usage of Insert/Update executors

reduce 200% memory usage for 500,000 rows * 8 bigint columns.

issue	difficulty	assignee	PRs
Reduce the memory usage of InsertExec/ UpdateExec #14355 Reduce the memory usage of the Insert executor	medium	@b41sh	#14568
Reduce the memory usage of InsertExec/ UpdateExec #14355 Reduce the memory usage of the Update executor	medium	help wanted	N/A

KR 3: Implement inline projection for executors

See #14428

• Support inline projection in planner @SunRunAway planner: prune columns again after predicate push down #14683
• Implement inline projection for Sort and TopN
• Join
  • Implement inline projection for Hash Join @SunRunAway executor: inline projection for hash join #14682
  • Implement inline projection for Merge Join
  • Implement inline projection for Index Merge Join
  • Implement inline projection for Index Hash Join
• Implement inline projection for Apply
• Implement inline projection Selection
• Implement inline projection for Limit

Objective 2: Polish & Bugfix

KR 1: Improve the JSON functionality

All issues are listed in #14486, here is a task list. In case you need help, please contact @qw4990

issue	difficulty	assignee	PRs
Support remaining JSON functions from MySQL 5.7 #7546 Implement the unsupported JSON function JSON_MERGE_PATCH	medium	help wanted	N/A
Support remaining JSON functions from MySQL 5.7 #7546 Implement the unsupported JSON function JSON_PRETTY	medium	help wanted	N/A
Call For Participation: finish the unsupported aggregate functions #7623 Implement the unsupported JSON function JSON_ARRAYAGG	medium	help wanted	N/A
Call For Participation: finish the unsupported aggregate functions #7623 Implement the unsupported JSON function JSON_OBJECTAGG	medium	@hg2990656	#11154
Support the JSON function JSON_STORAGE_SIZE #14488 Implement the unsupported JSON function JSON_STORAGE_SIZE	medium	help wanted	N/A
json_extract get wrong decimal when insert into table #12178 JSON_EXTRACT gets wrong decimal when insert into table	easy	help wanted	N/A
Invalid JSON value for CAST to DATE/TIME/DATETIME/TIMESTAMP from column json_extract for virtual column #13722 Invalid JSON value for CAST to DATE/TIME/DATETIME/TIMESTAMP from column json_extract for virtual column	easy	help wanted	N/A
json_object with repeat a long num is wrong. #9995 JSON_OBJECT handle long strings wrongly	easy	help wanted	N/A
Incorrect JSON object compare #10460 Compare json objects wrongly	easy	help wanted	N/A
Incorrect JSON float compare #10461 Compare json floats wrongly; difficulty	easy	help wanted	N/A
Insert Json loss accuracy #11386 Loss accuracy when inserting json records	easy	help wanted	N/A
JSON data type does not support custom types (i.e. MySQL types) #9988 Casting times to json cannot be compatible with MySQL	easy	help wanted	N/A
Incompatible with MySQL 8.0 about the CAST string to JSON #10390 Casting strings to json cannot be compatible with MySQL	easy	help wanted	N/A
JSON_EXTRACT fails to cast as bool #12233 JSON_EXTRACT fails to cast as bool	easy	help wanted	N/A
Convert json to uint different from mysql #11489 Convert json to uint different from mysql	easy	help wanted	N/A
Incorrect GROUP BY for JSON values #10467 Incorrect GROUP BY for JSON values	easy	help wanted	N/A
Incompatible issue related to JSON functions #13710 Incorrect results when processing JSON_EXTRACT(...) IN (...)	easy	help wanted	N/A

KR 2: #14332 Refine the implementation of IndexMergeReader

issue	difficulty	assignee	PRs
refine the IPC in indexMergeReaderExecutor #14333 Refine the IPC between partialWorker and indexMergeProcessWorker	easy	@sduzh	#14381
make RuntimeStats and feedback correct for IndexMergeReaderExecutor #14334 make RuntimeStats and feedback correct for IndexMergeReaderExecutor	easy	help wanted	N/A
use WithRecovery in IndexMergeReaderExecutor instead of recover #14335 use WithRecovery in IndexMergeReaderExecutor instead of recover	easy	@SeaRise	#14550
start workers in Next instead of Open for IndexMergeReaderExecutor #14336 start workers in Next instead of Open for IndexMergeReaderExecutor	easy	help wanted	N/A
reuse the result chk in IndexMergeReaderExecutor #14337 reuse the result chk in IndexMergeReaderExecutor	easy	help wanted	N/A
send a batch of handles in indexMergeTableScanWorker #14338 send a batch of handles in indexMergeTableScanWorker	easy	help wanted	N/A
#TODO Produce an article to introduce the IndexMergeReader	easy	@hailanwhu, @sduzh, @XuHuaiyu	N/A

KR 3: Fix null in subquery problem

• #13743 push down conditions error
• #13749 correlated select items should return NULL if the outer table is empty
• #13389 in subquery does not distinguish NULL
• #8844 Semi Join should be NULL-Aware
• #12065 wrong result and panic when joining dual table
• #12074 anti semi join error
• Executor, Planner: convert in_subquery to hash joins when in_subquery just involves one column #13972 improve in subquery performance

KR 4: #14392 Improve memory management for executors project/31

issue	difficulty	assignee	PRs
Proposal: a better way to track memory for chunks. #14358 Proposal: a better way to track memory for chunks	medium	help wanted	N/A
#TODO Use external storage for Index Merge Join	hard	help wanted	N/A
Implement grace hash join when build input is bigger than the available memory #14675 implement grace hash join for Hash Join to avoid OOM of hash table	hard	help wanted	N/A
make the memory tracing in HashAggExec more accurate #14705 make the memory tracing in HashAggExec more accurate	medium	help wanted	N/A
disk-based partialResultMap for HashAggExec #14413 Use external storage for Hash Aggregation	hard	help wanted	N/A
Disk-based distinct operation #14395 Use external storage for Distinct	hard	help wanted	N/A
Disk-based sort executor #12431 Use external storage for Sort	medium	@wshwsh12	#13718, #14279
#TODO Use external storage for Window Function	medium	help wanted	N/A
#TODO Compare between row-based and vectorized external storage format for different executors	hard	help wanted	N/A

KR 5: Investigate and implement Adaptive Join

• #14408 Investigate and propose Adaptive Join, difficulty: hard, assignee: help wanted

KR 6: Support the Partition memory table

• #13905 Support the INFORMATHIN_SCHEMA. Partition memory table like MySQL. difficulty: medium, assignee: @tangwz, @qw4990, deadline: 2020/02/10; infoschema: show information about table partitions in information_schema.PARTITIONS #14347

Objective 3: Support more features

KR 1: Support SELECT INTO OUTFILE syntax

• #11112 Support SELECT INTO OUTFILE. difficulty: medium, assignee: @tsthght, @qw4990, deadline: 2020/02/31
• #722 Let our parser support the SELECT INTO OUTFILE syntax. difficulty: medium, assignee: @tsthght, @qw4990, deadline: 2020/03/31

KR 2: Support to load configs from PD and update dynamically

• #13660 Support to load configs from PD and update dynamically. difficulty: hard, assignee: @qw4990, deadline: 2020/01/31; config: introduce a config client to support load configs from PD online #14303, proposal: dynamic configuration change #13660

[qw4990]

@SunRunAway Please add this #14705 to the list at your convenience.

[zz-jason]

close due to stale.