*: REORGANIZE PARTITION (#38535) #41096

* mockkv: make pk id = -1 if no primary key column is used (pingcap#38443) * ddl: support modify column on partitioned table (pingcap#38302) close pingcap#38297 * planner: revise isnullRejected check for `And` and `OR` (pingcap#38430) close pingcap#38304 Co-authored-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Song Gao <disxiaofei@163.com>

…| tidb-test=pr/2044 (pingcap#38508) ref pingcap#15000, ref pingcap#38535

ref pingcap#38535

* br: add retry for azblob read file (pingcap#39650) close pingcap#39649 * copr: fix data race in the mppIterator (pingcap#39772) close pingcap#39771 * testkit: set GOMAXPROCS in test (pingcap#39766) * statistics: support download history stats from stats_history (pingcap#39701) * *: Support Failed-Login Tracking and Temporary Account Locking (pingcap#39322) close pingcap#38938 * ddl: mark the writes from delete-only and drop them on merge (pingcap#39796) close pingcap#39731 * fix: update ioutil func&pruning case (pingcap#39798) * dumpling: meter network usage (pingcap#39774) close pingcap#39674 * staleread: fix flaky test TestStaleReadProcessorWithExecutePreparedStmt (pingcap#39805) close pingcap#39804 * domain: fix data race in the MockInfoCacheAndLoadInfoSchema (pingcap#39802) close pingcap#39801 * *: upgrade rules_go (pingcap#39797) * *: Add backfill job related tables and operations (pingcap#39616) close pingcap#37122 * ttl: implement ttl job schedule framework (pingcap#39472) close pingcap#39471 * *: stable tiflash test in ddl (pingcap#39825) close pingcap#39831 * planner: fix index prefix matching (pingcap#39377) close pingcap#14581 * parser: add cast to array and member of support (pingcap#39820) close pingcap#39821 * fix: update ioutil.readall (pingcap#39850) * planner: add three fields to statement summary table (pingcap#39828) ref pingcap#39199 * ddl: fix unexpect fail when create expression index (pingcap#39822) close pingcap#39784 * ttl: ouput TTL comment format when show create table (pingcap#39853) close pingcap#39852 * dumpling: add enable paging for tidb server (pingcap#39791) close pingcap#39790 * planner: support explain format='cost_trace' on model ver2 (pingcap#39840) ref pingcap#35240 * ttl, variables: add system variables for ttl job schedule (pingcap#39727) close pingcap#39725 * *: fix slow query nil pointer panic when get session for TTL (pingcap#39865) close pingcap#39864 * ttl: Add metrics for TTL (pingcap#39849) close pingcap#39848 * statistics: support gc outdated historical stats (pingcap#39788) * planner: rename `general plan cache` to `non-prepared plan cache` (pingcap#39873) * planner: rename 'tidb_enable_general_plan_cache' and 'tidb_general_plan_cache_size' to 'tidb_enable_non_prepared_plan_cache' and 'tidb_non_prepared_plan_cache_size' * planner: binding from cluster statements summary (pingcap#39737) ref pingcap#39199 * ttl: modify default value of `tidb_ttl_delete_batch_size` to 100 (pingcap#39878) close pingcap#39877 * ddl: close lightning writers after the import is complete (pingcap#39879) close pingcap#39800 * store/copr: handle region error from client (pingcap#39838) ref pingcap#39361 * ttl: fix scan workers shrink (pingcap#39860) close pingcap#39859 * docs: fix typos (pingcap#39870) * domain: support dump sql meta in plan replayer (pingcap#39863) close pingcap#39883 * SlowLog: Update `Wait_TS` field in slow log for every sql (pingcap#39843) close pingcap#39713 * session: fix a bug for InitDDLJobTables (pingcap#39861) ref pingcap#39854 * ttl: reschedule scan tasks after update task state (pingcap#39891) close pingcap#39890 * ttl: TTL support split scan ranges for some primary key with more than one columns (pingcap#39896) close pingcap#39895 * *: add dependabot (pingcap#39229) * expression: support `json_overlaps` function (pingcap#39867) ref pingcap#39866 * executor, planner: `InitLogger` before `CreateMockStore` in tests to avoid data race (pingcap#39900) close pingcap#39892 * statistics: add source column for stats_meta_hisotry table (pingcap#39835) * ttl: fix ttl txn will not return error when rolling back (pingcap#39918) close pingcap#39917 * expression: close recordset (pingcap#39899) * br: make more sane variables of ticking timeout (pingcap#39887) close pingcap#39620 * expression: support `member of` function (pingcap#39880) ref pingcap#39866 * br: fix err overwrite (pingcap#39935) close pingcap#39934 * ttl: add log in the whole lifecycle of a job (pingcap#39931) close pingcap#39930 * ttl: use optimistic transaction with 0 retry limit for ttl (pingcap#39924) close pingcap#39923 * ttl: periodically update state for a job in heartbeat (pingcap#39939) close pingcap#39938 * parser: support syntax set binding status by sql digest (pingcap#39515) ref pingcap#39199 * *: lock row keys during merging back from temp index (pingcap#39936) close pingcap#39929 * ttl: initialize notifyStateCh in ttl job manager (pingcap#39956) close pingcap#39955 * domain, statistics: periodically update stats healthy distribution (pingcap#39739) close pingcap#39349 * ttl: enable 1pc for ttl delete (pingcap#39958) close pingcap#39957 * docs/design: format the name of 'Adding-Index-Acceleration' RFC. (pingcap#39952) * *: fix auto analyze worker crash when killed by global memory limit (pingcap#39946) close pingcap#39947 * doc: add a proposal for foreign key (pingcap#35982) * domain: Support plan replayer continus capture (pingcap#39926) * expression, json: fix cast json as string with shorter length (pingcap#39970) close pingcap#39963 * sessionctx, executor: fix failure of reading MaxPreparedStmtCount (pingcap#39736) close pingcap#39735 * resourcemanger: create cpu monitor (pingcap#39962) ref pingcap#39657 * *: prevent cursor read from being cancelled by GC (pingcap#39950) close pingcap#39447 * dumpling: supports foreign key limitations (pingcap#39914) close pingcap#39913 * build(deps): bump golang.org/x/net from 0.2.0 to 0.4.0 (pingcap#39996) * planner: fix pessimistic-auto-commit not working for point plans (pingcap#39993) close pingcap#39928 * fix: ebs volume restore failure since incorrect unfinished volumes (pingcap#39975) close pingcap#39974 * planner, json: restore cast flag after substitute column in cast (pingcap#39997) close pingcap#39806 * staleread, session: internal write request should be accepted with external ts (pingcap#39967) close pingcap#39966 * *: revert 38302, forbid modify column on partition table (pingcap#39991) ref pingcap#39915, ref pingcap#39922 * executor: close recordset again (pingcap#40010) * ddl: retry prepare RPC when meets region error (pingcap#39834) close pingcap#39836 * metrics: add EMA cpu usage (pingcap#39995) close pingcap#40029 * executor: fix unstable TestMaxPreparedStmtCount (pingcap#40024) ref pingcap#39735 * *: support sample for compressed files for adjustment (pingcap#39680) ref pingcap#38514 * docs: fix 'enable_global_index' to 'enable-global-index' (pingcap#40009) * expression: introduce `cast(... as ... array)` in expression index (pingcap#39992) close pingcap#40045 * planner: fix the issue that NonPrep Cache cannot work with Explain well (pingcap#40044) close pingcap#40039 * ddl: fix waiting for wrong schema version if TiDB server restart (pingcap#39985) close pingcap#39981 * *: Fix issue 39999, used wrong column id list for checking partitions (pingcap#40003) close pingcap#39999 * oomtest: add whitelist for oom test to fix flaky test (pingcap#40055) close pingcap#40054 * *: extend skyline pruning diagnostic info and add extra warnings (pingcap#39894) ref pingcap#39893 * ddl: add privilege check when alter table add foreign key (pingcap#40051) close pingcap#40050 * planner: add more test cases for non-prep plan cache (pingcap#40060) * sessionctx: disable gc aware memory track (pingcap#40019) close pingcap#39971 * *: make auto-analyze killable by global memory limit (pingcap#39978) ref pingcap#39971, close pingcap#39994 * cpu: fix ticker to avoid close early (pingcap#40036) ref pingcap#40029 * build(deps): bump golang.org/x/time from 0.2.0 to 0.3.0 (pingcap#39912) * ddl: use latest ts to read record for adding index (pingcap#40081) close pingcap#40074 * executor: close recordset again (pingcap#40073) * planner: add more test cases for MPP hints (pingcap#39933) * planner: support set binding status by sql digest (pingcap#39517) ref pingcap#39199 * ddl: check the limitation when creating multi-valued index (pingcap#39818) close pingcap#40086 * sysvar: allow modifying 'tidb_allow_remove_auto_inc' when SEM is ON (pingcap#40083) close pingcap#38238 * *: fix issue of multi-schema change with foreign key (pingcap#40042) close pingcap#40037 * planner: check the `ignore-plan-cache` hint in insert-stmt (pingcap#40080) ref pingcap#39717, close pingcap#40079 * parser: support keep_order and no_keep_order hint (pingcap#39965) ref pingcap#39964 * *: filter particularly errors when `truncateAsWarning` is true (pingcap#40078) * metrics: add metrics for ema cpu metrics and GOGC (pingcap#40049) close pingcap#40092 * server: avoid reusing cached stmt ctx on cursor read (pingcap#40023) close pingcap#39998 * *: optimize mpp probe (pingcap#39932) close pingcap#39686 * statistics: fix the sync load sql did'nt running internally (pingcap#40087) close pingcap#39511 * server: support download plan replayer continues capture file (pingcap#40085) * *: support foreign key check/cascade when execute replace into statement (pingcap#40069) close pingcap#40068 * ddl: fix issue of add foreign key too slow in big table (pingcap#40112) close pingcap#40111 * ddl: fix issue of partition table with foreign key and add more test case (pingcap#40122) * parser: fix cast function restore (pingcap#40129) close pingcap#40130 * ddl: set context correctly in the setDDLLabelForDiagnosis (pingcap#40090) close pingcap#40099 * planner: move code of generating IndexMergePath to a separated file (pingcap#40134) * statistics: fix incorrect datetime value when loading stats (pingcap#40084) close pingcap#39336 * ttl: escape parameters for finish job sql (pingcap#40118) close pingcap#40115 * ddl: prohibit change name of partitioning column (pingcap#40147) close pingcap#40135 * lightning: disable foreign key checks (pingcap#40032) close pingcap#40027 * br: recursive split batch create table when restore failure on batch create table (pingcap#39829) close pingcap#39029 * store/copr: don't reset concurrency of analyze req to 2 (pingcap#40168) close pingcap#40162 * build(deps): bump golang.org/x/oauth2 from 0.2.0 to 0.3.0 (pingcap#40140) * domain: revise plan replayer process log (pingcap#40126) * expression: impl cast as array function (pingcap#40076) close pingcap#40128 * session: fix tidb_enable_gc_aware_memory_track after upgrade (pingcap#40173) ref pingcap#39971, close pingcap#40174 * restore: Fix prealloc (pingcap#40176) close pingcap#40177 * metrics: remove Keep Alive OPM metric (pingcap#40142) close pingcap#40143 * util: fix data race in the cpu test (pingcap#40154) close pingcap#40125 * executor: move some test into writetest (pingcap#40097) * planner: fix an unstable explain test case (pingcap#40199) ref pingcap#35240 * ddl: Refine the error message to compatible with MySQL when drop a partition table partition key column (pingcap#38740) close pingcap#38739 * executor: TiFlash supports stale read (pingcap#40048) close pingcap#40047 * ddl: ignore internal http client goroutine in leak tests (pingcap#40190) close pingcap#40183 * ddl: add more foreign key test case (pingcap#40052) close pingcap#40189 * *: add support for -initialize-sql-file on first bootstrap (pingcap#35625) close pingcap#35624 * *: fix PointGet will return an stale value when `tidb_enable_plan_replayer_capture` is set (pingcap#40197) close pingcap#40194 * planner: support converting `json_member_of` to IndexMerge to access MVIndex (pingcap#40175) ref pingcap#40191 * planner: record reasons when skipping Plan Cache (pingcap#40210) * session: Do not run telemetry loops when it's disabled in config (pingcap#40156) close pingcap#40155 * docs/design: REORGANIZE PARTITION design (pingcap#38246) ref pingcap#15000, ref pingcap#15002 * ddl: Disable RENAME COLUMN for partitioning columns | tidb-test=pr/2054 (pingcap#40151) close pingcap#40150 * dumpling: add foreign key e2e test (pingcap#40133) close pingcap#40227 * ttl: make the TTL compatible with dumpling, lightning and BR (pingcap#40216) close pingcap#40215 * ttl, domain: setup a customized session pool with stats collector (pingcap#40171) close pingcap#40109 * ddl: Support flashback cluster with ddl history (pingcap#40209) ref pingcap#40026 * lightning: add foreign key test (pingcap#40103) close pingcap#40226 * planner: refine planner code for disaggregated tiflash mode (pingcap#39813) close pingcap#39814 * docs: update tidb roadmap based on v6.5 (pingcap#40170) * planner: remove the unnecessary skip-plan-cache flag in StmtCtx (pingcap#40235) * ttl: fix change status sql argument (pingcap#40234) close pingcap#40231 * planner: update the plan cache strategy when expressions with parameters affect null-check (pingcap#40218) close pingcap#38205, close pingcap#40093 * go.mod: upgrade badger to 20221229114011 (pingcap#40236) close pingcap#39237 * executor: write multi-valued index (pingcap#40172) close pingcap#40207 * expression: implement `unhex` function pushdown to tiflash (pingcap#39898) ref pingcap#5112, ref pingcap/tiflash#5112 * planner: remove the unnecessary skip-plan-cache flag in StmtCtx (pingcap#40246) * restore: set config value to default when failed to get config from tikv (pingcap#36051) close pingcap#36053 * ddl: check default value again in updateColumnDefaultValue (pingcap#40230) close pingcap#40164 * docs: update roadmap (pingcap#40244) * *: remove the support of the old ddl framework (pingcap#39684) ref pingcap#39824 * lightning: check peers write stall when switch-mode is disabled (pingcap#40228) close pingcap#40163 * ddl, parser: make generated column and expression index same as MySQL (pingcap#39888) close pingcap#39826 * session: make plan replayer capture support prepared stmt (pingcap#40167) close pingcap#40161 * expression: implement `regexp_replace` function pushdown (pingcap#40267) close pingcap#40266 * planner: supply more test for binding from history (pingcap#40242) ref pingcap#39199 * ttl: disable ttl job when recover/flashback table/database/cluster (pingcap#40268) close pingcap#40265 * admin: impl admin check index for mv index (pingcap#40270) close pingcap#40272 * planner: refactor plan-cache UseCache flag (pingcap#40256) ref pingcap#36598 * test: fix data race in cast as array (pingcap#40277) close pingcap#40276 * planner: support converting `json_overlaps/contains` to IndexMerge to access MVIndex (pingcap#40195) ref pingcap#40191 * autoid_service: add unit test for the package (pingcap#40193) * executor: special handling is required when an "auto id out of range" error occurs in `insert ignore into ... on on duplicate ...` (pingcap#39847) close pingcap#38950 * util: gorotinue pool (pingcap#39872) close pingcap#38039 * metrics: add metrics for plan replayer and historical stats (pingcap#40271) * sysvar: fix circular dependency in rebuildSysVarCache leading to deadlock (pingcap#40283) close pingcap#40240 * ddl: fix flaky test TestGlobalVariablesOnFlashback (pingcap#40291) close pingcap#40287 * server, tidb-server: kill auto analyze when gracefully shutting down (pingcap#40284) close pingcap#40038 * planner: better coordination between the ignore_plan_cache() binding and plan-cache (pingcap#40280) close pingcap#40225 * store/copr: fix build batchCop in disaggregated tiflash mode (pingcap#40008) close pingcap#40035 * ddl: persist index info after changing backfill state (pingcap#40229) close pingcap#40217 * doc: update flashback doc (pingcap#40031) ref pingcap#40026 * ddl, parser: Implement the write-reorg state split task related functions, and the related interfaces of backfill worker (pingcap#39982) close pingcap#37123 * planner: support pushing down predicates to memory tables in prepared mode (pingcap#40262) close pingcap#39605 * planner: open the partial order prop push down for LIST PARTITION (pingcap#40290) close pingcap#40273 * br: fix bootstrap domain (pingcap#40307) close pingcap#40281 * br: add errch buf for checkpoint (pingcap#40166) close pingcap#40165 * planner: skip plan-cache for prepared queries with `INT in (Decimals...)` (pingcap#40312) close pingcap#40224 * statistics: support historical stats dump partition table (pingcap#40310) * *: update badger (pingcap#40331) * expression: enlarge timeout for test (pingcap#40332) * resourcemanger: add cpu scheduler (pingcap#39886) close pingcap#39657 * util: use go-deadlock to find deadlock (pingcap#40288) close pingcap#40293 * *: support `curdate()` as column's default value | tidb-test=pr/2057 (pingcap#40326) close pingcap#38356 * ttl: don't schedule ttl job when EnableTTLJob is off (pingcap#40336) close pingcap#40335 * *: add warnings to slow log and slow query table (pingcap#39884) ref pingcap#39893 * planner,infoschema,executor: Add tiflash fine grained shuffle support for hash join and aggregation (pingcap#40121) close pingcap#40123 * ddl: fill in original default for extra writable columns in batch insert (pingcap#40198) close pingcap#40192 * ddltest: update goleak and enable race (pingcap#40357) close pingcap#40358 * planner: let the planner can select IndexMerge on MVIndex without hint (pingcap#40347) ref pingcap#40191 * makefile: enable deadlock in the with-real-tikv (pingcap#40359) close pingcap#40293 * planner: support more types to use IndexMerge to access MVIndex (pingcap#40343) ref pingcap#40191 * log-backup: should set default value `gc.ratio-threshold` after pitr finished (pingcap#40141) close pingcap#40185 * domain: let plan replayer dump support recording stats mem status (pingcap#40334) * docs: add TiDB Forum link and adjust link order (pingcap#39868) * ddl: support online create multi-valued index (pingcap#40304) close pingcap#40337 * ttl: add `ttl_job_interval` attribute and remove `tidb_ttl_job_interval` variable (pingcap#40033) close pingcap#40028 * *: remove the support of the `amending transaction` (pingcap#39714) close pingcap#40381 * executor: reset the related session vars for both INSERT and REPLACE (pingcap#40354) close pingcap#40351 * *: fix static pruning partition table in disaggregated tiflash mode (pingcap#40238) close pingcap#40239 * planner: fix typo (pingcap#40367) * session: fix data race in the LazyTxn.LockKeys (pingcap#40350) close pingcap#40355 * build(deps): bump golang.org/x/crypto from 0.1.0 to 0.5.0 (pingcap#40379) * executor: fix data race at the ShowExec (pingcap#39817) close pingcap#39816, close pingcap#40295 * planner: fix panic during starting tidb-server if creating global binding for partition table (pingcap#40402) * fix Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Supports create/alter/drop resource group (pingcap#40188) ref pingcap#38825 * session: fix deadlock when init domain failed (pingcap#40409) close pingcap#40408 * executor: support insert ignore/duplicate replace into with unique multi-valued index (pingcap#40369) close pingcap#40207 * br: support reset_tiflash after ebs restoration (pingcap#40124) close pingcap#40208 * ttl: add ttl goroutine exit log (pingcap#40416) close pingcap#40415 * planner, executor: split the range for unsigned pk of partition table when limit is used (pingcap#40313) close pingcap#40309 * planner: add nil check when handling empty charset in `setUnionFlen` (pingcap#40429) close pingcap#40285 * planner: support keep_order and no_keep_order hint (pingcap#40089) ref pingcap#39964 * planner: unify the behavior of prepare/execute limit to mysql (pingcap#40360) ref pingcap#40219 * gpool: register gpool into resource manager (pingcap#40410) ref pingcap#40412 * ttl: Add `CommandClient` to trigger TTL job manually (pingcap#40346) close pingcap#40345 * ttl: make ttl's group in granfana collapse by default (pingcap#40444) * br: reset cloud_admin and root after ebs restoration (pingcap#39986) close pingcap#40418 * ddl: support displaying sub-job reorg type in admin show ddl (pingcap#40387) close pingcap#40386 * planner: support use IndexMerge to access composite MVIndex (pingcap#40426) ref pingcap#40191 * planner: add test cases for keep_order and no_keep_order hints (pingcap#40321) ref pingcap#39964 * statistics: change the upper bound of the out-of-range estimation to modify count | tidb-test=pr/2012 (pingcap#39011) close pingcap#39008 * docs: update CI build badages (pingcap#40324) close pingcap#40325 * planner: update the non-prep cacheable checker (pingcap#40446) close pingcap#40296 * expression, planner: exclude concat_ws from aggresive folding constant (pingcap#38383) close pingcap#36888 * keyspace: introduce keyspace conf and etcd path (pingcap#40269) ref pingcap#40425 * ddl: skip deleting the unmatched index key on merge (pingcap#40465) close pingcap#40464 * Merge pingcap/master and pingcap/feature/reorganize-partition * Updated CODEOWNERS for the feature branch * ci: add document about some task that will be triggered manually (pingcap#40459) * errno, store: fix ErrPDServerTimeout arguments (pingcap#40468) close pingcap#40439 * ttl: fix the TTL job reports error when primary key contains a column with type `ENUM` (pingcap#40457) close pingcap#40456 * planner: support using DNF to construct IndexMerge to access MVIndex (pingcap#40471) * fixup * fixup * fixup * executor: add partition table testcase for historical stats (pingcap#40453) * add partition table testcase * add partition table testcase * fix lint Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * ttl: forbid creating/altering a table with TTL options when pk contains float/double column (pingcap#40487) * ttl: forbid create/alter a table with TTL options when pk contains float/double column * format * update * update * update Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * planner: make MVIndex be compatible with the `use/force index` hint (pingcap#40495) * fixup * fixup * fixup * fixup * lightning: fix codecPDClient (pingcap#40496) Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com> Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> * ddl: let concurrent truncate on the same table depend on the previous one (pingcap#40501) * done Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com> * refine test Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com> Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * statistics: fix estimation error when ranges are too many and modify count is large (pingcap#40472) * fix * fmt * bazel update * update test result Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * executor: fix issue of KEY_COLUMN_USAGE table return wrong result about foreign key column (pingcap#40491) Signed-off-by: crazycs520 <crazycs520@gmail.com> Signed-off-by: crazycs520 <crazycs520@gmail.com> Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * sysvar: add switch for plan replayer capture using historical stats (pingcap#40492) * add switch * fix * fix * fix Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: upgrade go1.19.5 (pingcap#40483) Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com> Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * br: need init MDL variable when GetDomain (pingcap#40512) Signed-off-by: joccau <zak.zhao@pingcap.com> Signed-off-by: joccau <zak.zhao@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * ddl: avoid commit conflicts when updating/delete from mysql.tidb_ddl_reorg. (pingcap#38738) * Added test case * ddl fix pingcap#38669. The issue was that mysql.tidb_ddl_reorg table was updated by an inner transaction after the outer transaction started, which then made a commit conflict in the outer transaction, when it deleted the same row. * Fixed typo in comment * Added test case for pingcap#24427 * Disabled tests for CI testing * Revert "Disabled tests for CI testing" This reverts commit 17c28f3. * Revert "Revert "Disabled tests for CI testing"" This reverts commit 65c84d9. * removed test skips * Clean up the tidb_ddl_reorg entry after DDL is completed * Use a cleanup job afterwards instead. * Fixed test * Moved cleanup before asyncNotify * More detailed test failure log * Refined test error message * Injecting timoeut to get stack traces from CI * Updated Debug Dump on timeout * Delete mulitple entries in tidb_ddl_reorg if needed * Linting * Linting * Added CI debug logs * Linting + CI debugs * fixed CI debug * Try to cleanup also if job.State == synced * check for non-error of runErr instead of error... * Use a new session, instead of reusing worker.sess * Also handle case when job == nil * Removed CI debug logs * Misssed change session from w.sess to newly created sess * Improved TestConcurrentDDLSwitch and added CI debug logs * Always cleaning up all orphan mysql.tidb_ddl_reorg entries * linting * Also cleanup if job is nil * Updated TestModifyColumnReorgInfo + CI debug logs * more CI debug * refactored the cleanupDDLReorgHandle code * Added missing cleanup in handleDDLJobQueue * Removed debug panic * Code cleanup * Test updates * Debug cleanup * Cleaned up test after removal of old non-concurrent DDL code merge * Linting * always wrap changes to tidb_ddl_reorg in an own transaction + fixed some typos * Minimum fix * Always update reorg meta, not only on error * Issue is here :) * Fixed newReorgHandler * Wrapped more tidb_ddl_reorg changes into separate transactions * linting * Removed updateDDLReorgStartHandle * cleanups * Made runInTxn a method on *session, instead of normal function * Update test * Final touches * Removed duplicate test * CleanupDDLReorgHandles should only be called from HandleJobDone. * Variable rename * Renamed 'delete' variabel name * Updated test * small revert * Removed timeout debugging code * Simplified the cleanup to only start a new txn and not a new session * Reverted the change of GetDDLInfo Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Post merge fix Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com> Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com> Signed-off-by: crazycs520 <crazycs520@gmail.com> Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com> Signed-off-by: joccau <zak.zhao@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com> Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: Song Gao <disxiaofei@163.com> Co-authored-by: lastincisor <64511824+lastincisor@users.noreply.github.com> Co-authored-by: tangenta <tangenta@126.com> Co-authored-by: Bin <49082129+songzhibin97@users.noreply.github.com> Co-authored-by: Obliviate <756541536@qq.com> Co-authored-by: Lynn <zimu_xia@126.com> Co-authored-by: YangKeao <yangkeao@chunibyo.icu> Co-authored-by: Daniël van Eeden <git@myname.nl> Co-authored-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: fzzf678 <108643977+fzzf678@users.noreply.github.com> Co-authored-by: Hangjie Mo <mohangjie1995@gmail.com> Co-authored-by: 王超 <cclcwangchao@hotmail.com> Co-authored-by: Chunzhu Li <lichunzhu@stu.xjtu.edu.cn> Co-authored-by: Yuanjia Zhang <zhangyuanjia@pingcap.com> Co-authored-by: you06 <you1474600@gmail.com> Co-authored-by: raymonder jin <rayjingithub@gmail.com> Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com> Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: Yifan Xu <30385241+xuyifangreeneyes@users.noreply.github.com> Co-authored-by: bb7133 <bb7133@gmail.com> Co-authored-by: Xiaoju Wu <wuxiaoju@pingcap.com> Co-authored-by: crazycs <crazycs520@gmail.com> Co-authored-by: zyguan <zhongyangguan@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: fengou1 <85682690+fengou1@users.noreply.github.com> Co-authored-by: wjHuang <huangwenjun1997@gmail.com> Co-authored-by: mychoxin <mychoxin@gmail.com> Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> Co-authored-by: CbcWestwolf <1004626265@qq.com> Co-authored-by: hackersean <hackersean@users.noreply.github.com> Co-authored-by: Yujie Xia <xiayjchn@gmail.com> Co-authored-by: jiyfhust <jiyfhust@163.com> Co-authored-by: hehechen <chentongli@pingcap.com> Co-authored-by: Morgan Tocker <mtocker@squareup.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> Co-authored-by: guo-shaoge <shaoge1994@163.com> Co-authored-by: Ran <huangran.alex@gmail.com> Co-authored-by: baishen <baishen2009@gmail.com> Co-authored-by: lance6716 <lance6716@gmail.com> Co-authored-by: xzhangxian1008 <xzhangxian@foxmail.com> Co-authored-by: tiancaiamao <tiancaiamao@gmail.com> Co-authored-by: Jk Xu <54522439+Dousir9@users.noreply.github.com> Co-authored-by: djshow832 <zhangming@pingcap.com> Co-authored-by: Yiding Cui <winoros@gmail.com> Co-authored-by: yibin <huyibin@pingcap.com> Co-authored-by: Zak Zhao <57036248+joccau@users.noreply.github.com> Co-authored-by: luzizhuo <496521310@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: wuhuizuo <wuhuizuo@126.com> Co-authored-by: ystaticy <y_static_y@sina.com> Co-authored-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>