-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fedora编译be模块失败 #28
Comments
Not a problem anymore, using docker dev environment instead |
Closed
stdpain
pushed a commit
to stdpain/incubator-doris
that referenced
this issue
Jul 5, 2021
2. Support the count aggregate_function 3. Some code refactor
HappenLee
added a commit
to HappenLee/incubator-doris
that referenced
this issue
Jul 15, 2021
2. Support the count aggregate_function 3. Some code refactor
HappenLee
added a commit
to HappenLee/incubator-doris
that referenced
this issue
Sep 7, 2021
2. Support the count aggregate_function 3. Some code refactor
3 tasks
3 tasks
3 tasks
This was referenced Jun 7, 2022
3 tasks
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jun 20, 2022
Co-authored-by: deardeng <565620795@qq.com> Squashed commit of the following: commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>
3 tasks
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this issue
Jul 7, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>
3 tasks
This was referenced Jul 19, 2022
This was referenced Jul 22, 2022
3 tasks
5 tasks
morrySnow
pushed a commit
that referenced
this issue
Jun 6, 2023
…e on clause (#20211) Add the rule for checking the join node in `analysis/CheckAnalysis.java` file. When we check the join node, we should check its' on clause. If there are some subquery expression, we should throw exception. Before this PR ``` mysql> select a.k1 from baseall a join test b on b.k2 in (select 49); ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: nul ``` After this PR ``` mysql> select a.k1 from baseall a join test b on b.k2 in (select 49); ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Not support OnClause contain Subquery, expr:k2 IN (INSUBQUERY) (LogicalOneRowRelation ( projects=[49 AS `49`#28], buildUnionNode=true )) ```
xiaokang
pushed a commit
that referenced
this issue
Jun 6, 2023
…e on clause (#20211) Add the rule for checking the join node in `analysis/CheckAnalysis.java` file. When we check the join node, we should check its' on clause. If there are some subquery expression, we should throw exception. Before this PR ``` mysql> select a.k1 from baseall a join test b on b.k2 in (select 49); ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: nul ``` After this PR ``` mysql> select a.k1 from baseall a join test b on b.k2 in (select 49); ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Not support OnClause contain Subquery, expr:k2 IN (INSUBQUERY) (LogicalOneRowRelation ( projects=[49 AS `49`#28], buildUnionNode=true )) ```
SWJTU-ZhangLei
added a commit
to SWJTU-ZhangLei/incubator-doris
that referenced
this issue
Jul 25, 2023
…cloud-dev (630da93 20230712) (apache#2065) This is a HUGE and UNSTABLE merge. It may cause compile error, crash/core-dump, and death. ``` 20230606 20230707 20230712 61b71de3a6 . 630da93a1b . . . doris-2.0 --------o---------------------.-----------------.-------- \__ . . \ . . dev-merge-2.0 ------o------o---o--o-o-------o----------o------.-------- / / \ . / ___/ \___ . / / \. selectdb-cloud-dev --o--------------------o------------------------o-------- ``` ============================================================= * [feature](function) add json->operator convert to json_extract (#19899) * [Fix](single replica load) fix indices_size key not found core (#20047) * [fix](fe)ordering exprs should be substituted in the same way as select part (#20091) * [improvement](exec) Refactor the partition sort node to send data in pipeline mode (#20128) before: the node will wait to retrieve all data from child, then send data to parent. now: for data from child that does not require sorting, it can be sent to parent immediately. * [regression](test) fix test case failed in pipeline mode (#20139) * [improvement](community) simplify the pr template and modify pr labeler (#20127) * [typo](doc)spark load add task timeout parameter #20115 * [refactor-WIP](TaskWorkerPool) add specific classes for PUSH, PUBLIC_VERION, CLEAR_TRANSACTION tasks (#19822) * [fix](ldap) fix ldap related errors (#19959) 1. fix ldap user show grants return null pointer exception; 2. fix ldap user show databases return no authority db; 3. ldap authentication supports catalog level; * [enhance](FileWriter)enhance s3 file writer bvar to avoid adding abort bytes (#20138) * don't add each time upload or it would add aborted bytes * alloca memory * [Enhancement](alter inverted index) Improve alter inverted index performance with light weight add or drop inverted index (#19063) * [typo](docs)Best usage document correction. #20142 * [refactor-WIP](TaskWorkerPool) add specific classes for ALTER_TABLE, CLONE, STORAGE_MEDIUM_MIGRATE task (#20140) * [Improve](data-type) Clean datatype uselesscode (#20145) * fix struct_export out data * delete useless code with data type * [chore](toolchain) change doris default toolchain to clang (#20146) GCC is very slow during build and link. Change to clang as we discussed many times. Co-authored-by: yiguolei <yiguolei@gmail.com> * [Fix](load)Make insert timeout accurate in `show load` statistics (#20068) * [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested (#19947) We encountered one confusing situation where buffered reader were trapped in one endless loop when calling readat. Then we found out that it was all due to the return data size is less than requested. As the following picture shows, the actual data size is about 2M, and when we called readat it only retrieved about 1MB. * [security] Don't print password in BaseController (#18862) * [fix](tablet_manager_lock) fix create tablet timeout #20067 (#20069) * [fix] (clone) fix drop biggest version replica during reblance step (#20107) * add check for rebalancer choose deleted replica * impr a compare * [enhancement](load) add some profile items for load (#20141) * [Improvement](topn) prevent memory usage of key topn increasing unlimited (#19978) * [Feature](Nereids) support advanced materialized view (#19650) Increase the functionality of advanced materialized view This feature already supported by legacy planner with PR #19650 This PR implement it in Nereids. This PR implement the features as below: 1. Support multiple columns in aggregate function. eg: select sum(c1 + c2) from t1; 2. Supports complex expressions. eg: select abs(c1), sum(abc(c1+1) + 1) from t1; TODO: 1. Support adding where in materialized view * [refactor](exec) replace the single pointer with an array of 'conjuncts' in ExecNode (#19758) Refactoring the filtering conditions in the current ExecNode from an expression tree to an array can simplify the process of adding runtime filters. It eliminates the need for complex merge operations and removes the requirement for the frontend to combine expressions into a single entity. By representing the filtering conditions as an array, each condition can be treated individually, making it easier to add runtime filters without the need for complex merging logic. The array can store the individual conditions, and the runtime filter logic can iterate through the array to apply the filters as needed. This refactoring simplifies the codebase, improves readability, and reduces the complexity associated with handling filtering conditions and adding runtime filters. It separates the conditions into discrete entities, enabling more straightforward manipulation and management within the execution node. * [fix](executor) Fixed an error with cast as time. #20144 before mysql [(none)]>select cast("10:10:10" as time); +-------------------------------+ | CAST('10:10:10' AS TIMEV2(0)) | +-------------------------------+ | 00:00:00 | +-------------------------------+ after mysql [(none)]>select cast("10:10:10" as time); +-------------------------------+ | CAST('10:10:10' AS TIMEV2(0)) | +-------------------------------+ | 10:10:10 | +-------------------------------+ In the past, we supported this syntax. mysql [(none)]>select cast("2023:05:01 13:14:15" as time); +------------------------------------------+ | CAST('2023:05:01 13:14:15' AS TIMEV2(0)) | +------------------------------------------+ | 13:14:15 | +------------------------------------------+ However, "10:10:10" is also a valid datetime. mysql [(none)]>select cast("10:10:10" as datetime); +-----------------------------------+ | CAST('10:10:10' AS DATETIMEV2(0)) | +-----------------------------------+ | 2010-10-10 00:00:00 | +-----------------------------------+ So here, the order of parsing has been adjusted. * [Fix](inverted index) fix memeory leak when inverted index writer do not finish correctly (#20028) * [Fix](inverted index) fix memeory leak when inverted index writer do not finish correctly * [Update](inverted index) use smart pointer to avoid memeory leak * [Chore](format) code format --------- Co-authored-by: airborne12 <airborne12@gmail.com> * [fix](dynamic_partition) fix dynamic partition not work when drop and recover olap table (#19031) when olap table is dynamic partition enable, if drop and recover olap table, the table should be added to DynamicPartitionScheduler again --------- Co-authored-by: caiconghui1 <caiconghui1@jd.com> * [Feature](agg_state) support agg_state combinators (#19969) support agg_state combinators state/merge/union * [Chore](build) add non-virtual-dtor, remove no-embedded-directive/no-zero-length-array (#20118) add non-virtual-dtor, remove no-embedded-directive/no-zero-length-array * [Fix](multi catalog)Fix Iceberg table missing column unique id bug (#20152) This pr is to fix the bug introduced by PR #19909 The bug failed to set column unique id for iceberg table, which will cause the query result for iceberg table are all NULL. ``` mysql> select * from iceberg_partition_lower_case_parquet limit 1; +------+------+------+---------+ | k1 | k2 | k3 | city | +------+------+------+---------+ | NULL | NULL | NULL | Beijing | +------+------+------+---------+ 1 row in set (0.60 sec) ``` After fix: ``` mysql> select * from iceberg_partition_lower_case_parquet limit 1; +------+------+------+---------+ | k1 | k2 | k3 | city | +------+------+------+---------+ | 1 | k2_1 | k3_1 | Beijing | +------+------+------+---------+ 1 row in set (0.35 sec) ``` * [typo](doc)correct the misspelled word and the improper word (#20149) * [Conf](decimalv3) enable decimalv3 by default * [feat](stats) delete data size stat and Made task timeout configurable (#20090) 1. Delete the stats for data size, since it would cost too much time but useless 2. Make task time out configurable since when it's common to analyze a quite huge table that the default 10 min is not suitable * [FIX](mysql_writer) fix mysql output binary object works (#20154) * fix struct_export out data * fix mysql writer output with binary true * [fix](partial update) use correct tablet schema for rowset writer in publish task (#20117) * [typo](config)Remove FE config max_conn_per_user (#20122) --------- Co-authored-by: Yijia Su <suyijia@selectdb.com> * [Improve](performance) introduce SchemaCache to cache TabletSchame & Schema (#20037) * [Improve](performance) introduce SchemaCache to cache TabletSchame & Schema 1. When the system is under high-concurrency load with wide table point queries, the frequent memory allocation and deallocation of Schema become evident system bottlenecks. Additionally, the initialization of TabletSchema and Schema also becomes a CPU hotspot.Therefore, the introduction of a SchemaCache is implemented to cache these resources for reuse. 2. Make some variables wrapped with std::unique<unique_ptr> Performance: | 状态 | QPS | 平均响应时间 (avg) | P99 响应时间 | |------------------|-----|------------------|-------------| | 开启 SchemaCache | 501 | 20ms | 34ms | | 关闭 SchemaCache | 321 | 31ms | 61ms | * handle schema change with schema version * remove useless header * rebase * [Chore](gensrc) remove gen_vector_functions.py #20150 * [fix](function) Fix VcompoundPred execute const column #20158 recurrent: ./run-regression-test.sh --run -suiteParallel 1 -actionParallel 1 -parallel 1 -d query_p0/sql_functions/window_functions select /*+ SET_VAR(query_timeout = 600) */ subq_0.`c1` as c0 from (select ref_1.`s_name` as c0, ref_1.`s_suppkey` as c1, ref_1.`s_address` as c2, ref_1.`s_address` as c3 from regression_test_query_p0_sql_functions_window_functions.tpch_tiny_supplier as ref_1 where (ref_1.`s_name` is NULL) or (ref_1.`s_acctbal` is not NULL)) as subq_0 where (subq_0.`c3` is NULL) or (subq_0.`c2` is not NULL) reason: FunctionIsNull and FunctionIsNotNull execute returns a const column, but their VectorizedFnCall::is_constant returns false, which causes problems with const handling when VCompoundPred::execute. This pr converts const column to full column in VCompoundPred execute. In the future, there will be a more thorough solution to such problems. * [Bug](function) fix equals implements not judge order by elements of function call expr (#20083) fix equals implements not judge order by elements of function call expr #19296 * [Enhancement](Nerieds) add switch for developing Nereids DML (#20100) * [Chore](build) adjust some compile diagnostic (#20162) * [BUG]storage_min_left_capacity_bytes default value has integer overflow #19943 * [fix](DECIMALV3) Fix the error in DECIMALV3 when explicitly casting. (#19926) before mysql [test]>select cast(1 as DECIMALV3(16, 2)) / cast(3 as DECIMALV3(16, 2)); +-----------------------------------------------------------+ | CAST(1 AS DECIMALV3(16, 2)) / CAST(3 AS DECIMALV3(16, 2)) | +-----------------------------------------------------------+ | 0.00 | +-----------------------------------------------------------+ mysql [test]>select * from divtest; +------+------+ | id | val | +------+------+ | 3 | 5.00 | | 2 | 4.00 | | 1 | 3.00 | +------+------+ mysql [test]>select cast(1 as decimalv3(16,2)) / val from divtest; +-------------------------------------+ | CAST(1 AS DECIMALV3(16, 2)) / `val` | +-------------------------------------+ | 0 | | 0 | | 0 | +-------------------------------------+ after mysql [test]>select cast(1 as DECIMALV3(16, 2)) / cast(3 as DECIMALV3(16, 2)); +-----------------------------------------------------------+ | CAST(1 AS DECIMALV3(16, 2)) / CAST(3 AS DECIMALV3(16, 2)) | +-----------------------------------------------------------+ | 0.33 | +-----------------------------------------------------------+ mysql [test]>select cast(1 as decimalv3(16,2)) / val from divtest; +-------------------------------------+ | CAST(1 AS DECIMALV3(16, 2)) / `val` | +-------------------------------------+ | 0.250000 | | 0.200000 | | 0.333333 | +-------------------------------------+ This is because in the previous code, the constant 1.000 would be transformed into 1. remove "ReduceType * [typo](docs) fix fqdn doc error (#20171) * [Feature](inverted index) add parser_mode properties for inverted index parser (#20116) We add parser mode for inverted index, usage like this: ``` CREATE TABLE `inverted` ( `FIELD0` text NULL, `FIELD1` text NULL, `FIELD2` text NULL, `FIELD3` text NULL, INDEX idx_name1 (`FIELD0`) USING INVERTED PROPERTIES("parser" = "chinese", "parser_mode" = "fine_grained") COMMENT '', INDEX idx_name2 (`FIELD1`) USING INVERTED PROPERTIES("parser" = "chinese", "parser_mode" = "coarse_grained") COMMENT '' ) ENGINE=OLAP ); ``` * [fix](p0 regression)Update hive docker test case result data (#20176) Doris updated array type output format, using double quote for Strings. Before, it was using single quote. So we need to update the case out file using double quote. * [Fix](multi-catalog) Fix parquet bugs of #19758 'replace the single pointer with an array of 'conjuncts' in ExecNode'. (#20191) Fix some parquet reader bugs which introduced by #19758 'replace the single pointer with an array of 'conjuncts' in ExecNode'. * [Fix](multi-catalog) fix all nested type test which introduced by #19518(support insert-only transactional table). (#20194) Fix `qt_nested_types_orc` in `test_tvf_p2` which introduced by #19518(support insert-only transactional table). ### Test case error `qt_nested_types_orc` in `test_tvf_p2` ``` select count(array0), count(array1), count(array2), count(array3), count(struct0), count(struct1), count(map0) from hdfs( "uri" = "hdfs://172.21.16.47:4007/catalog/tvf/orc/all_nested_types.orc", "format" = "orc", "fs.defaultFS" = "hdfs://172.21.16.47:4007") ``` **Error Message:** errCode = 2, detailMessage = (172.21.0.101)[INTERNAL_ERROR]Wrong data type for colum 'struct1' * [feat](optimizer) Support CTE reuse (#19934) Before this PR, new optimizer would inline CTE directly. However in many scenario a CTE could be referenced many times, such as in TPC-DS tests, for these cases materialize the result sets of CTE and reuse it would significantly agument performance. In our tests on tpc-ds related sqls, it would improve the performance by up to almost **4 times** than before. We introduce belowing plan node in optimizer 1. CTEConsumer: which hold a reference to CTEProducer 2. CTEProducer: Plan defined by CTE stmt 3. CTEAnchor: the father node of CTEProducer, a CTEProducer could only be referenced from corresponding CTEAnchor's right child. A CTEConsumer would be converted to a inlined plan if corresponding CTE referenced less than or equal `inline_cte_referenced_threshold` (it's a session variable, by default is 1). For SQL: ```sql EXPLAIN REWRITTEN PLAN WITH cte AS (SELECT col2 FROM t1) SELECT * FROM t1 WHERE (col3 IN (SELECT c1.col2 FROM cte c1)) UNION ALL SELECT * FROM t1 WHERE (col3 IN (SELECT c1.col2 FROM cte c1)); ``` Rewritten plan before this PR: ``` +------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +------------------------------------------------------------------------------------------------------------------------------------------------------+ | LogicalUnion ( qualifier=ALL, outputs=[col1#14, col2#15, col3#16], hasPushedFilter=false ) | | |--LogicalJoin[559] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(col3#6 = col2#8)], otherJoinConjuncts=[] ) | | | |--LogicalProject[551] ( distinct=false, projects=[col1#4, col2#5, col3#6], excepts=[], canEliminate=true ) | | | | +--LogicalFilter[549] ( predicates=(__DORIS_DELETE_SIGN__#7 = 0) ) | | | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | | | +--LogicalProject[555] ( distinct=false, projects=[col2#20 AS `col2`#8], excepts=[], canEliminate=true ) | | | +--LogicalFilter[553] ( predicates=(__DORIS_DELETE_SIGN__#22 = 0) ) | | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | | +--LogicalProject[575] ( distinct=false, projects=[col1#9, col2#10, col3#11], excepts=[], canEliminate=false ) | | +--LogicalJoin[573] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(col3#11 = col2#13)], otherJoinConjuncts=[] ) | | |--LogicalProject[565] ( distinct=false, projects=[col1#9, col2#10, col3#11], excepts=[], canEliminate=true ) | | | +--LogicalFilter[563] ( predicates=(__DORIS_DELETE_SIGN__#12 = 0) ) | | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | | +--LogicalProject[569] ( distinct=false, projects=[col2#24 AS `col2`#13], excepts=[], canEliminate=true ) | | +--LogicalFilter[567] ( predicates=(__DORIS_DELETE_SIGN__#26 = 0) ) | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | +------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` After this PR ``` +------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +------------------------------------------------------------------------------------------------------------------------------------------------------+ | LogicalUnion ( qualifier=ALL, outputs=[col1#14, col2#15, col3#16], hasPushedFilter=false ) | | |--LOGICAL_CTE_ANCHOR#-1164890733 | | | |--LOGICAL_CTE_PRODUCER#-1164890733 | | | | +--LogicalProject[427] ( distinct=false, projects=[col2#1], excepts=[], canEliminate=true ) | | | | +--LogicalFilter[425] ( predicates=(__DORIS_DELETE_SIGN__#3 = 0) ) | | | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | | | +--LogicalJoin[373] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(col3#6 = col2#8)], otherJoinConjuncts=[] ) | | | |--LogicalProject[370] ( distinct=false, projects=[col1#4, col2#5, col3#6], excepts=[], canEliminate=true ) | | | | +--LogicalFilter[368] ( predicates=(__DORIS_DELETE_SIGN__#7 = 0) ) | | | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | | | +--LOGICAL_CTE_CONSUMER#-1164890733#1038782805 | | +--LogicalProject[384] ( distinct=false, projects=[col1#9, col2#10, col3#11], excepts=[], canEliminate=false ) | | +--LogicalJoin[382] ( type=LEFT_SEMI_JOIN, markJoinSlotReference=Optional.empty, hashJoinConjuncts=[(col3#11 = col2#13)], otherJoinConjuncts=[] ) | | |--LogicalProject[379] ( distinct=false, projects=[col1#9, col2#10, col3#11], excepts=[], canEliminate=true ) | | | +--LogicalFilter[377] ( predicates=(__DORIS_DELETE_SIGN__#12 = 0) ) | | | +--LogicalOlapScan ( qualified=default_cluster:test.t1, indexName=t1, selectedIndexId=42723, preAgg=ON ) | | +--LOGICAL_CTE_CONSUMER#-1164890733#858618008 | +------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` * Revert "[fix](DECIMALV3) Fix the error in DECIMALV3 when explicitly casting. (#19926)" (#20204) This reverts commit 8ca4f9306763b5a18ffda27a07ab03cc77351e35. * [Bug](segment iterator) remove DCHECK for block row count (#20199) CHECK rows count of block at segment iterator is not ready when `enable_common_expr_pushdown` * [Improvement](runtimefilter) Build bloom filter according to the exact build size for IN_OR_BLOOM_FILTER (#20166) * [Improvement](runtimefilter) Build bloom filter according to the exact build size for IN_OR_BLOOM_FILTER * [Enhance](array function) add support for DecimalV3 for array_enumerate_uniq() (#17724) * [doc](fix)Modified the description about trino #20174 * [typo](docs) fix oceanbase jdbc catalog error (#20197) * [runtimeFilter](nereids) use runtime filter default size for debug purpose (#20065) use rf default size for debug * [enhance](match) Support match query without inverted index (#19936) * [Fix](multi-catalog) Fix q03 in `text_external_brown` regression test by handling correctly when text converter parsing error. (#20190) Issue Number: close #20189 Fix `q03` in `text_external_brown` regression test by handling correctly when text converter parsing error. * [regression](decimalv3) Fix output for P1 regression (#20213) * [fix](test) fix p2 broker load (#20196) * [fix](session-variable) fix set global var on non-master FE return error (#20179) * [fix](catalog) fix create catalog with resource replay issue and kerberos auth issue (#20137) 1. Fix create catalog with resource replay bug. If user create catalog using `create catalog hive with resource xxx`, when replaying edit log, there is a bug that resource may be dropped, causing NPE and FE will fail to start. In this PR, I add a new FE config `disallow_create_catalog_with_resource`, default is true. So that `with resource` will not be allowed, and it will be deprecated later. And also fix the replay bug to avoid NPE. 2. Fix issue when creating 2 hive catalogs to connect with and without kerberos authentication. When user create 2 hive catalogs, one use simple auth, the other use kerberos auth. The query may fail with error like: `Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.` So I add a default property for hive catalog: `"ipc.client.fallback-to-simple-auth-allowed" = "true"`. Which means this property will be added automatically when user creating hive catalog, to avoid such problem. 3. Fix calling `hdfsExists()` issue When calling `hdfsExists()` with non-zero return code, should check if it encounters error or is file not found. 3. Some code refactor Avoid import `org.apache.parquet.Strings` * [fix](Nereids) filter should not push through union to OneRowRelation (#20132) ## Problem summary When we want to push the filter through the union. We should check whether the union's children are `OneRowRelation` or not. If there are some `OneRowRelation`, we shouldn't push down the filter to that part Before this PR ``` mysql> select * from (select 1 as a, 2 as b union all select 3, 3) t where a = 1; +------+------+ | a | b | +------+------+ | 1 | 2 | | 3 | 3 | +------+------+ 2 rows in set (0.01 sec) ``` After this PR ``` mysql> select * from (select 1 as a, 2 as b union all select 3, 3) t where a = 1; +------+------+ | a | b | +------+------+ | 1 | 2 | +------+------+ 1 row in set (0.38 sec) ``` * [Feature-WIP](inverted index) support phrase for inverted index writer (#20193) * [opt](Nereids) refactor the PartitionTopN (#20102) Do some small refactoring for the `PartitionTopN` and also address the left comment in #18784 * [Feature](Inverted index) add MATCH_ PHRASE query (#20156) * support query queue (#20048) support query queue (#20048) * [Enchancement](execute) make assert_cast can output derived class name (#20212) before: F0530 11:02:41.989699 1154607 assert_cast.h:54] Bad cast from type:doris::vectorized::IDataType const* to doris::vectorized::DataTypeAggState const* after: F0530 11:24:28.390286 1292475 assert_cast.h:46] Bad cast from type:doris::vectorized::DataTypeNullable* to doris::vectorized::DataTypeAggState const* * [Improve](CI)Check PR approve status (#20172) After discussion in the doris community @apache/doris-committers , we limit the PR to be merged only after at least two people approve it.↳ We can try to run it for a while first, and if everyone gives good feedback, we can use this as a mandatory check. Since the merge must be approved by at least one committer, we only need to judge whether there are two approves, and we don't need to care about the identity of the approve. When there is a request change, if the other party is a committer, the committer dismiss is required when merging, which is enforced by github, so we don't need to care. * [Feature](compaction) wip: single replica compaction (#19237) Currently, compaction is executed separately for each backend, and the reconstruction of the index during compaction leads to high CPU usage. To address this, we are introducing single replica compaction, where a specific primary replica is selected to perform compaction, and the remaining replicas fetch the compaction results from the primary replica. The Backend (BE) requests replica information for all peers corresponding to a tablet from the Frontend (FE). This information includes the host where the replica is located and the replica_id. By calculating hash(replica_id), the replica with the smallest hash value is responsible for executing compaction, while the remaining replicas are responsible for fetching the compaction results from this replica. The compaction task producer thread, before submitting a compaction task, checks whether the local replica should fetch from its peer. If it should, the task is then submitted to the single replica compaction thread pool. When performing single replica compaction, the process begins by requesting rowset versions from the target replica. These rowset_versions are then compared with the local rowset versions. The first version that can be fetched is selected. * [test](regression) add regression test from materialized slot bug (#20207) The test query includes the conversion of string types to other types, and the processing of materialized columns for nested subqueries, which is the regression test for bug fix(#18783) * [Fix](multi catalog, nereids)Fix text file required slot bug (#20214) required_slots in TFileScanRangeParams params for external hive table may be updated after FileQueryScanNode finalize. For text file, we need to use the origin required_slots in params so that the list could be updated later. Otherwise, query text file may get the following error: [INTERNAL_ERROR]Unknown source slot descriptor, slot_id=3 * [feature](nereids) support the rewrite rule for push-down filter through sort (#20161) Support the rewrite rule for push-down filter through sort. We can directly push-down the filter through sort without any conditions check. Before this PR: ``` mysql> explain select * from (select * from t1 order by a) t2 where t2.b > 2; +-------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#2] | | b[#3] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 3:VSELECT | | | predicates: b[#3] > 2 | | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: a[#0] | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | | | 1:VTOP-N | | | order by: a[#2] ASC | | | offset: 0 | | | | | 0:VOlapScanNode | | TABLE: default_cluster:test.t1(t1), PREAGGREGATION: ON | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | +-------------------------------------------------------------+ 30 rows in set (0.06 sec) ``` After this PR: ``` mysql> explain select * from (select * from t1 order by a) t2 where t2.b > 2; +-------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | a[#2] | | b[#3] | | PARTITION: UNPARTITIONED | | | | VRESULT SINK | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: a[#0] | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | | | 1:VTOP-N | | | order by: a[#2] ASC | | | offset: 0 | | | | | 0:VOlapScanNode | | TABLE: default_cluster:test.t1(t1), PREAGGREGATION: ON | | PREDICATES: b[#1] > 2 | | partitions=0/1, tablets=0/0, tabletList= | | cardinality=1, avgRowSize=0.0, numNodes=1 | +-------------------------------------------------------------+ 28 rows in set (0.40 sec) ``` * [Enhencement](JDBC Catalog) refactor jdbc catalog insert logic (#19950) This PR refactors the old way of writing data to JDBC External Table & JDBC Catalog, mainly including the following tasks 1. Continuing the work of @BePPPower 's PR #18594, changing the logic of splicing Inster sql to operating off-heap memory and using preparedStatement.set to write data logic to complete 2. Supplement the support written by largeint type, mainly to adapt to Java.Math.BigInteger, which uses binary operations 3. Delete the splicing SQL logic in the JDBC External Table & JDBC Catalog related written code ToDo: Binary type,like bit,binary, blob... Finally, special thanks to @BePPPower , @AshinGau for his work Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com> * [regression](p0) fix test for `array_enumerate_uniq` (#20231) * [Chore](log) Remove some verbose log && Change log level (#20236) * [feature-wip](workload-group) Support setting user default workload group (#20180) Issue Number: close #xxx SET PROPERTY 'default_workload_group' = 'group_name'; * [fix](regression)Update external Brown test case out file. #20232 Update external Brown test case out file to match the new precision. * [chore](third-party) Bump the version of hadoop_libs (#20250) Fix the issues with the workflow Build Third Party Libraries. See https://github.com/apache/doris-thirdparty/actions/runs/5109407220/jobs/9184234534 * [docs](spark-doris-connector): modify the link of spark-doris-connector (#20159) * [Enhancement](merge-on-write) optimize bloom filter for primary key index (#20182) * [Bug](runtimefilter) Fix waiting for runtime filter (#20155) * [fix](nereids) like function's nullable property should be PropagateNullable (#20237) * [Fix](dynamic-partition) Try to avoid setting a zero-bucket-size partition. (#20177) A fallback to avoid BE crash problem when partition's bucket size is 0, but not resolved. * [chore](arm) support build with hadoop libhdfs on arm (#20256) hadoop-3.3.4.3-for-doris already support build on arm * [fix](pipeline) Enable pipeline explicitly in the plan shape check cases. (#20221) enable pipeline explicitly in tpcds plan shape check * [Fix](Planner)fix incorrect pattern when format pattern contains %x%v (#19994) * [Enhancement] Change Create Resource Group Grammar (#20249) * [enhancement](ldap) Support refresh ldap cache (#20183) Support refreshing ldap cache: refresh ldap all; refresh ldap; refresh ldap for user1; Support for caching non-existent ldap users. When logging in with a doris user that does not exist in the Ldap service after ldap is enabled, avoid accessing the ldap service every time in scenarios such as show databases; that require a lot of authentication. * [Fix](Nereids) fold constant result is wrong on functions relative to timezone (#19863) * [improvement](bitmap) Using set to store a small number of elements to improve performance (#19973) Test on SSB 100g: select lo_suppkey, count(distinct lo_linenumber) from lineorder group by lo_suppkey; exec time: 4.388s create materialized view: create materialized view customer_uv as select lo_suppkey, bitmap_union(to_bitmap(lo_linenumber)) from lineorder group by lo_suppkey; select lo_suppkey, count(distinct lo_linenumber) from lineorder group by lo_suppkey; exec time: 12.908s test with the patch, exec time: 5.790s * [opt](nereids) generate in-bloom filter if target is local for pipeline mode (#20112) update in-filter usage in pipeline mode: 1. if the target is local, we use in-bloom filter. Let BE choose in or bloom according to actual distinctive number 2. set default runtime_filter_max_in_num to 1024 * [fix](match_phrase) Fix the inconsistent query result for 'match_phrase' after creating index without support_phrase property (#20258) if create inverted index without support_phrase property, remaining the match_phrase condition to filter by match function. * [deps](aws) upgrade to 1.9.272 to fix non-compliant RFC3986 encoding (#20252) * [Bug](memtable) fix a bug occurred when we were inserting data into duplicate table without keys (#20233) * [fix](load_profile) fix rows stat and add close_wait in sink (#20181) * [Fix](Nereids) bitmap type should not be used in comparison predicate (#19807) When using nereids, if we use compare operator of bitmap type, an analyze exception need to be throwed. like: select id from (select BITMAP_EMPTY() as c0 from expr_test) as ref0 where c0 = 1 order by id Which c0 in subq0 is a bitmap type, this scenario is not supported right now. * [fix](checksum) delete predicates might be inconsistent with rowset readers in checksum task (#20251) The BlockReader capture rowsets and init delete_handler in different place. If there is a base compaction, it may result in obtaining inconsistent delete handlers. Therefore, place these two operations under the same lock. * [fix](tvf) s3 tvf specify region and s3.region params failed (#19921) * [Enhancement](merge-on-write) Performance optimization of calculations of delete bitmap between segments (#20153) 1. Use heap sort to find duplicated keys between segments and update the delete-bitmap. The old implementation traversed all keys in all segments, used each key to search for duplicates in earlier segments, and then marked them for deletion. 2. Trick: Each time the heap top is popped as a key1, the new heap top is key2, allowing for jumping directly from key1 to key2 instead of advancing iteratively. 3. Effect: This technique works well when there are many segments within the same rowset and the imported data is relatively ordered. * [refactor](dynamic table) Make segment_writer unaware of dynamic schema, and ensure parsing is exception-safe. (#19594) 1. make ColumnObject exception safe 2. introduce FlushContext and construct schema at memtable flush stage to make segment independent from dynamic schema 3. add more test cases * [fix](regression-test) add jdbc timeout (#20228) In some cases ( or bugs), doris may returned query to jdbc, but jdbc can not recognized what doris sent back, so hanged. To fix this, add a timeout of 30 minutes to jdbc connection. * [fix][regression-test] set timeout of curl in regression test to avoid hanged when be crashed. (#20222) Currently in regression-test, when a be crash, because curl does not set a timeout, suite-thread will get stuck. To solve this, encapsulate the call to be into a function, set the timeout uniformly, and avoid getting stuck * [fix](nereids)(planner) case when should return NullLiteral when all case result is NullLiteral (#20280) * [bug](parse) fix can't create aggregate column with agg_state (#20235) fix can't create aggregate column with agg_state * [pipeline](load) support pipeline load (#20217) * fix fe meta upgrade error (#20291) Co-authored-by: yiguolei <yiguolei@gmail.com> * [Fix](Nereids) fix some insert into select bugs (#20052) fix 3 bugs: 1. failed to insert into a table with mv. ```sql create table t ( id int, c1 int, c2 int, c3 int ) duplicate key(id) distributed by hash(id) buckets 4 create materialized view k12s3m as select id, sum(c1), max(c3) from t group by id; insert into t select -4, -4, -4, 'd'; ``` insert will rise exception because mv column is not handled. now we will add a target column and value as defineExpr. 2. failed to insert into a table with not all the columns. ```sql insert into t(c1, c2) select c1, c2 from t ``` and t(id ukey, c1, c2, c3), will insert too many data, we fix it by change the output partitions. 3. failed to insert into a table with complex select. the select statement has join or agg, fix the bug by the way similar to the one at 2nd bug. * [fix](multi catalog)Fix nereids planner text format include extra column index bug (#20260) Nereids planner include all columns index in TFileScanRangeParams, this may cause the column projection incorrect for text format table. Because csv reader use the column index position to split a line. Extra column index will cause get wrong split result. This PR is to reset the column index after Projection, remove the useless column index. * [fix](regression) regression test test_bitmap_filter_nereids could not run (#20293) * [feature](decimal)support cast rounding half up and div precision increment in decimalv3. (#19811) * [improvement](Nereids): limit Memo groupExpression size. (#20272) * [Improve](Scan) add a session variable to make scan run serial (#20220) Parallel scanning can result in some read amplification, for example, select * from xx where limit 1 actually requires only one row of data. However, due to parallel scanning of multiple tablets, read amplification occurs, leading to performance bottlenecks in high-concurrency scenarios. This PR Adding a SessionVariable to enforce serial scanning can help mitigate this issue. * [P2](test) Fix P2 output (#20311) * [Fix](Nereids) Fix function test case unstable by adding order by (#20295) Nereids function case do not have a order by clause, so the result will be unstable, so order by is added to ensure stability. * [fix](docs)Correct the year and month format placeholder to lower case (#20210) * [Bug](exec) push down no group by agg min cause error result (#20289) sql """ CREATE TABLE t1_int ( num int(11) NULL, dgs_jkrq bigint(20) NULL ) ENGINE=OLAP DUPLICATE KEY(num) COMMENT 'OLAP' DISTRIBUTED BY HASH(num) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "storage_format" = "V2", "light_schema_change" = "true", "disable_auto_compaction" = "false", "enable_single_replica_compaction" = "false" ); """ sql """insert into t1_int values(1,1),(1,2),(1,3),(1,4),(1,null);""" qt_sql """ select min(dgs_jkrq) from t1_int; """ get the error result:4 after change we get the right result:1 * [fix](regression-test) fix multi-thread problem of regression-test #20322 * [bug](udaf) fix java-udaf test case failed with decimal (#20315) java-udaf have some test case with decimal will be failed in P0, because the decimal of scale is not set correctly * [enhancement](publish) print detailed info for failed publish (#20309) * [feature-wip](duplicate_no_keys) Add some test cases of all the duplicate tables in test case tpcds_sf100_without_key_p2 and make them duplicate tables without keys (#20332) * [fix](regression-test) variable's scope returned by curl (#20347) * [fix](planner)Fix missing kw for workload #20319 1 add usage docment for Workload Group query queue; 2 Fix missing KW for workload, this may cause create workload group failed. * [feature-wip](duplicate-no-keys) schame change support for duplicate no keys (#19326) * [fix](nereids)dphyper join reorder may cache wrong project list for project node (#20209) * [fix](nereids)dphyper join reorder may cache wrong project list for project node * [pipeline](rpc) support closure reuse in pipeline exec engine (#20278) * [Bug](runtime filter) fix NPE if runtime filter has no target (#20338) * [typo](docs)Correct the getting started document (#20245) * [enhancement](struct-type)support comment for struct field (#20200) support comment for struct field * [Fix](Nereids) should not gather data when sink (#20330) * [Fix](multi-catalog) fix oss access issue with aws s3 sdk (#20287) * [Docs](inverted index) update docs for inverted index parser_mode and match_phrase support (#20266) * [Profile](exec) Remove unless profile in pipeline exec engine (#20337) * [Bug](pipeline) Fix memory leak if query is canceled caused by memory limit (#20316) * [typo](docs) fix release note 2.0 zh url (#20320) * [refactor](stats) Persist status of analyze task to FE meta data (#20264) 1. In the past, we use a BE table named `analysis_jobs` to persist the status of analyze jobs/tasks, however there are many flaws such as, if BE crashed analyze job/task would failed however the status of analyze job/task couldn't get updated. 2. Support `DROP ANALYZE JOB [job_id]` to delete analyze job 3. Support `SHOW ANALYZE TASK STATUS [job_id] ` to get the task status of specific job 4. Restrict the execute condition of auto analyze, only when the last execution of auto analyze job finished a while ago could be executed again 5. Support analyze whole DB * [performance](load) support parallel memtable flush for unique key tables (#20308) * [fix](olap) deletion statement with space conditions did not take effect (#20349) Deletion statement like this: delete from tb where k1 = ' '; The rows whose k1's value is ' ' will not be deleted. * [Feature](array-functions)improve array functions for array_last_index (#20294) Now we just support array_first_index for lambda input , but no array_last_index * [pipeline](fix) rm github_token, no need for it (#20360) * [Improve](json-array) Support json array with nereids bool (#20248) Support json array with nereids bool now : ``` set enable_nereids_planner=true; mysql> SELECT json_array(1, "abc", NULL, TRUE, '10:00:00'); +----------------------------------------------+ | json_array(1, 'abc', NULL, TRUE, '10:00:00') | +----------------------------------------------+ | [1,"abc",null,false,"10:00:00"] | +----------------------------------------------+ 1 row in set (0.02 sec) ``` nereids boolean is "true"/"false" is not '0' /'1' , so we always get false * [enhancement](txn) print commit backends when commit fails (#20367) Print commit backends when a commit fails. * [fix](Nereids) forbid unexpected expression on filter and fix two more bugs (#20331) fix below bugs: 1. not check filter's expression, aggregate function, grouping scalar function and window expression should not appear in filter 2. show not change nullable of aggregate function when it is window function in window expression 3. bitmap and other metric types should not appear in order by or partition by of window expression * [Optimize](Function) Add fast path for col like '%%' or col like '%' or regexp '\\.*' (#20143) Add fast path for col like '%%' or col like '%' or regexp '\\.*' (1) like about 34% speed up when use count() test support col like '%%' , col like '%', col not like '%%' , col not like '%' (2) regexp about 37% speed up when use count() test support col regexp '\\.', col not regexp '\\.' Q1: select count() From hits where url like '%'; Q2: select count() From hits where url regexp '\\.*'; * [fix](nereids) add fragment id on all PhysicalRelation (#20371) fix "cannot find fragment id for scan" exception * [chore](third-party) Bump the version of hadoop_libs (#20369) Bump the version of hadoop_libs to build HDFS related libraries only. * [chore](function) Refactor FunctionSet Initialization for Better Maintainability and Compilation Success (#20285) In this PR, I have refactored the initialization of the FunctionSet. Previously, all the functions were in one large method which led to the generation of Java code that was too long. This posed a problem for the compiler, as the length of the method exceeded the limit imposed by the Java compiler. To resolve this issue and improve the readability and manageability of our code, I have categorized these functions by type, and created dedicated initialization methods for each type. As such, our code is now not only more readable and understandable, but also each method is of a length that is acceptable to the compiler and can be compiled successfully. Moreover, this change makes it easier for us to add new functions as we can directly locate the right category and add new functions there. This is a significant change aimed at enhancing the maintainability and scalability of our code, while ensuring that our code can be successfully compiled. * [Enhancement](tvf) Backends tvf supports authentication (#20333) Add authentication for backends tvf. * [typo](docs) Update the `help create` command display (#20357) * [refactor](jdbc catalog) Refactor the JdbcClient code (#20109) This PR does the following: 1. This PR is a substantial refactor of the JDBC client architecture. The previous monolithic JDBC client has been refactored into an abstract base class `JdbcClient`, and a set of database-specific subclasses (e.g., `JdbcMySQLClient`, `JdbcOracleClient`, etc.), and the JdbcClient required config, abstract into an object. This allows for improved modularity, easier addition of support for new databases, and cleaner, more maintainable code. This change is backward-compatible and does not affect existing functionality. 2. As a result of client refactoring, OceanBaseClient can automatically recognize the mode of operation as MySQL or Oracle, so we cancel the oceanbase_mode property in the Jdbc Catalog, but due to the cancellation of the property, When creating a single OceanBase Jdbc Table, the table type needs to be filled in as oceanbase(mysql mode) or oceanbase_oracle(oracle_mode). The above work is a change in the usage behavior, please note. 3. For the PostgreSQL Jdbc Catalog, I did two things: 1. The adaptation to MATERIALIZED VIEW and FOREIGN TABLE is added 2. Fixed reading jsonb, which had been incorrectly changed to json in a previous PR 4. fix some jdbc catalog test case 5. modify oceanbase jdbc doc And,Thanks @wolfboys for the guidance * [Docs](docs) Update BE http documents (#17604) * [fix](match) fix match query with compound predicates return -6003 (#20361) * [fix](workload-group) fix incorrect memoryLimitPercent value (#20377) * [fix](Nereids) should not inherit child's limit and offset when generate exchange node (#20373) in legacy planner, when we new exchange, it inherit its child's limit and offset. but in Nereids, we should not do this. because if we need set limit or offset, we will set it manually. In this PR, we use a new ctor of ExchangeNode to ensure not set limit or offset unexpected. * [fix](match query) fix array column match query failed without inverted index (#20344) * [Enchancement](function) optimize for padding function && add string length check on string op (#20363) * [Bug](schema-change) make test_dup_mv_schema_change more stable #20379 make test_dup_mv_schema_change more stable * [fix](regression) test_partial_update_with_row_column (#20279) * [DOCS](data-types) remove old types (#20375) * [typo](docs)clearly describe the rename syntax (#20335) * [fix](dynamic_partition) fix dead lock when modify dynamic partition property for olap table (#20390) Co-authored-by: caiconghui1 <caiconghui1@jd.com> * [opt](MergedIO) optimize merge small IO, prevent amplified read (#20305) Optimize the strategy of merging small IO to prevent severe read amplification, and turn off merged IO when file cache enabled. Adjustable parameters: ``` // the max amplified read ratio when merging small IO max_amplified_read_ratio=0.8 // the min segment size file_cache_min_file_segment_size = 1048576 ``` * [improvement](inverted index) skip write index on load and generate index on compaction (#20325) * [fix](inverted index) fix transaction id changed when light index change (#20302) * [typo](doc)Add a demo of export minio (#20323) * [docs](workload-group) add user binding workload group docs (#20382) * [docs](load-balancing):delete duplicate sentences and improve the documentation description (#20297) * [Doc](statistics) supplement stats doc (regression test and automatic collection) (#20071) * [docs](auth) forbid 127.0.0.1 passwd free login (#19096) * [typo](doc)Update stream-load-manual.md (#20277) Modify the sequential label * [typo](doc)Update compilation-general.md (#20262) Add some explanations about docker run parameter * [typo](doc)Update compilation-general.md (#20261) Add some explanations about docker run parameter * [typo](doc)Update runtime-filter.md (#20292) * [typo](doc)Remove the description of the BE configuration 'serialize_batch' which has been removed (#20163) * [typo](doc) Fixed typos in hive.md (#19457) * [fix](community) fix PR template (#20400) * [pipeline](opt) Opt fragment instance prepare performance by thread pool (#20399) * [Fix](lazy_open) fix lazy open commit info lose (#20404) * [build](scripts) modify build-for-release.sh (#20398) * [Fix](Planner)fix cast date/datev2/datetime to float/double return null. (#20008) * [fix](Nereids) give clean error message when there are subquery in the on clause (#20211) Add the rule for checking the join node in `analysis/CheckAnalysis.java` file. When we check the join node, we should check its' on clause. If there are some subquery expression, we should throw exception. Before this PR ``` mysql> select a.k1 from baseall a join test b on b.k2 in (select 49); ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: nul ``` After this PR ``` mysql> select a.k1 from baseall a join test b on b.k2 in (select 49); ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Not support OnClause contain Subquery, expr:k2 IN (INSUBQUERY) (LogicalOneRowRelation ( projects=[49 AS `49`#28], buildUnionNode=true )) ``` * [opt](Nereids) perfer use datev2 / datetimev2 in date related functions (#20224) 1. update all date related functions' signatures order. 1.1. if return value need to be compute with time info, args with datetimev2 at the top of the list, followed by datev2, datetime and date 1.2. if return value need to be compute with only date info, args with datev2 at the top of list, followed by datetimev2, date and datetime 2. Priority for use datev2, if we must cast date to datev2 or datetime/datetimev2 * [fix](dynamic partition) partition create failed after alter distributed column (#20239) This pr fix following two problems: Problem1: Alter column comment make add dynamic partition failed inside issue #10811 create table with dynamic partition policy; restart FE; alter distribution column comment; alter dynamic_partition.end to trigger add new partition by dynamic partition scheduler; Then we got the error log, and the new partition create failed. dynamic add partition failed: errCode = 2, detailMessage = Cannot assign hash distribution with different distribution cols. default is: [id int(11) NULL COMMENT 'new_comment_of_id'], db: default_cluster:example_db, table: test_2 Problem2: rename distributed column, make old partition insert failed. inside #20405 The key point of the reproduce steps is restart FE. It seems all versions will be affected, include master and lts-1.1 and so on. * [fix](memory) Fix query memory tracking #20253 The memory released by the query end is recorded in the query mem tracker, main memory in _runtime_state. fix page no cache memory tracking Now the main reason for the inaccurate query memory tracking is that the virtual memory used by the query is sometimes much larger than the actual memory. And the mem hook counts virtual memory. * [fix](nereids) select with specified partition name is not work as expected (#20269) This PR is to fix the select specific partition issue, certain codes related to this feature were accidentally deleted. * [Optimize](function) Optimize locate function by compare across strings (#20290) Optimize locate function by compare across strings. about 90% speed up test by sum() * [Enchancement](Agg State) storage function name and result is nullable in agg state type (#20298) storage function name and result is nullable in agg state type * [fix](Nereids): fix filter can't be pushdown unionAll (#20310) * [Feature](Nereids) support update unique table statement (#20313) * [feature](profile)Add the filtering info of the in filter in profile #20321 image Currently, it is difficult to obtain the id of in filters,so, the some in filters's id is -1. * [feature](planner)(nereids) support user defined variable (#20334) Support user-defined variables. After this PR, we can use `set @a = xx` to define a user variable and use it in the query like `select @a`. the changes of this PR: 1. Support the grammar for `set user variable` in the parser. 2. Add the `userVars` in `VariableMgr` to store the user-defined variables. 3. For the `set @a = xx`, we will store the variable name and its value in the `userVars` in `VariableMgr`. 4. For the `select @a`, we will get the value for the variable name in `userVars`. * [improve](nereids)derive analytics node stats (#20340) 1. derive analytic node stats, add support for rank() 2. filter estimation stats derive updated. update row count of filter column. 3. use ColumnStatistics.orginal to replace ColumnStatistics.orginalNdv, where ColumnStatistics.orginal is the column statisics get from TableScan. TPCDS 70 on tpcds_sf100 improved from 23sec to 2 sec This pr has no performance downgrade on other tpcds queries and tpch queries. * [fix](nereids) avg size of column stats always be 0 (#20341) it takes lot of effort to compute the avgSizeByte for col stats. we use schema information to avoid compute actual average size * [fix](stats) skip forbid_unknown_col_stats check for invisible column and internal db (#20362) 1. skip forbidUnknownColStats check for in-visible columns 2. use columsStatistics.isUnknown to tell if this stats is unknown 3. skip unknown stats check for internal schema * [Fix](Nereids) Fix duplicated name in view does not throw exception (#20374) When using nereids, if we have duplicated name in output of view, we need to throw an exception. A check rule was added in bindExpression rule set * [fix](load) in strict mode, return error for insert if datatype convert fails (#20378) * [fix](load) in strict mode, return error for load and insert if datatype convert fails Revert "[fix](MySQL) the way Doris handles boolean type is consistent with MySQL (#19416)" This reverts commit 68eb420cabe5b26b09d6d4a2724ae12699bdee87. Since it changed other behaviours, e.g. in strict mode insert into t_int values ("a"), it will result 0 is inserted into table, but it should return error instead. * fix be ut * fix regression tests * [fix](nereids) change defaultConcreteType function's return value for decimal (#20380) 1. add default decimalv2 and decimalv3 for NullType 2. change defaultConcreteType of decimalv3 to this * [performance](load) improve memtable sort performance (#20392) * [fix][refactor](backend-policy)(compute) refactor the hierarchy of external scan node and fix compute node bug #20402 There should be 2 kinds of ScanNode: OlapScanNode ExternalScanNode The Backends used for ExternalScanNode should be controlled by FederationBackendPolicy. But currently, only FileScanNode is controlled by FederationBackendPolicy, other scan node such as MysqlScanNode, JdbcScanNode will use Mix Backend even if we enable and prefer to use Compute Backend. In this PR, I modified the hierarchy of ExternalScanNode, the new hierarchy is: ScanNode OlapScanNode SchemaScanNode ExternalScanNode MetadataScanNode DataGenScanNode EsScanNode OdbcScanNode MysqlScanNode JdbcScanNode FileScanNode FileLoadScanNode FileQueryScanNode MaxComputeScanNode IcebergScanNode TVFScanNode HiveScanNode HudiScanNode And previously, the BackendPolicy is the member of FileScanNode, now I moved it to the ExternalScanNode. So that all subtype ExternalScanNode can use BackendPolicy to choose Compute Backend to execute the query. All all ExternalScanNode should implement the abstract method createScanRangeLocations(). For scan node like jdbc scan node/mysql scan node, the scan range locations will be selected randomly from compute node(if preferred). And for compute node selection. If all scan nodes are external scan nodes, and prefer_compute_node_for_external_table is set to true, the BE for this query will only select compute nodes. * [fix](sequence) value predicates shouldn't be push down when has sequence column (#20408) * (fix)[sequence] value predicates shouldn't be push down when has sequence column * add case * [Fix] (tablet) fix tablet queryable set (#20413) (#20414) * [fix](conf) fix fe host in doris-cluster.conf #20422 * [fix](workload-group) fix workload group non-existence error (#20428) * Fix query hang when using queue (#20434) * [fix](execution) result_filter_data should be filled by 0 when can_filter_all is true (#20438) * [fix](Nereids) throw NPE when sql cannot be parsed by all planner (#20440) * [bug](jdbc) fix trino date/datetime filter (#20443) When querying Trino's JDBC catalog, if our WHERE filter condition is k1 >= '2022-01-01', this format is incorrect. In Trino, the correct format should be k1 >= date '2022-01-01' or k1 >= timestamp '2022-01-01 00:00:00'. Therefore, the date string in the WHERE condition needs to be converted to the date or timestamp format supported by Trino. * [fix](load) fix generate delete bitmap in memtable flush (#20446) 1. Generate delete bitmap for one segment at a time. 2. Generate delete bitmap before segment compaction. Fix #20445 * [fix](executor)Fix duplicate timer and add open timer #20448 1 Currently, Node's total timer couter has timed twice(in Open and alloc_resource), this may cause timer in profile is not correct. 2 Add more timer to find more code which may cost much time. * [improvement](column reader) lazy load indices (#20456) Currently when reading column data, all types of indice are read even if they are not actually used, this PR implements lazy load of indices. * [enhancement](profile) add build get child next time (#20460) Currently, build time not include child(1)->get next time, it is very confusing during shared hash table scenario. So that I add a profile. --------- Co-authored-by: yiguolei <yiguolei@gmail.com> * [fix](regression) fix export file test cases (#20463) * [Fix](WorkloadGroup)Fix query queue nereids bug #20484 * [fix](Nereids) join condition not extract as conjunctions (#20498) * [fix](log) publish version log is printed too frequently (#20507) * fix fe part compile error v1 * fix fe part compile error v2 * fix fe part compile error v3 * fix fe…
starocean999
pushed a commit
to starocean999/incubator-doris
that referenced
this issue
Jul 26, 2023
yiguolei
added a commit
to yiguolei/incubator-doris
that referenced
this issue
Dec 12, 2023
3 tasks
3 tasks
yiguolei
pushed a commit
that referenced
this issue
Jul 18, 2024
…out range of _buf (#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 #5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 #6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 #7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 #10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 #11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 #12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 #13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 #18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 #19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 #22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 #23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 #28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) #29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 #5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
dataroaring
pushed a commit
that referenced
this issue
Jul 19, 2024
…out range of _buf (#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 #5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 #6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 #7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 #10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 #11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 #12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 #13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 #18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 #19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 #22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 #23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 #28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) #29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 #5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
yiguolei
pushed a commit
that referenced
this issue
Aug 10, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 #5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 #6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp #7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 #8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 #9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 #10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 #11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 #12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 #13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 #14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 #15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 #16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 #17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 #18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 #19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 #20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 #21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 #22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 #23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h #24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 #25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 #26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 #27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 #28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 #29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 #30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 #31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 #32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 #33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 #34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 #38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 #39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 #40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
mrhhsg
added a commit
to mrhhsg/doris
that referenced
this issue
Aug 11, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 apache#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 apache#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 apache#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 apache#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
dataroaring
pushed a commit
that referenced
this issue
Aug 11, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 #5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 #6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp #7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 #8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 #9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 #10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 #11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 #12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 #13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 #14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 #15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 #16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 #17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 #18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 #19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 #20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 #21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 #22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 #23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h #24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 #25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 #26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 #27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 #28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 #29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 #30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 #31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 #32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 #33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 #34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 #38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 #39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 #40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
yiguolei
pushed a commit
that referenced
this issue
Aug 12, 2024
## Proposed changes pick #39191 Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 #5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 #6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp #7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 #8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 #9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 #10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 #11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 #12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 #13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 #14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 #15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 #16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 #17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 #18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 #19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 #20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 #21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 #22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 #23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h #24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 #25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 #26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 #27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 #28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 #29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 #30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 #31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 #32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 #33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 #34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 #38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 #39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 #40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this issue
Aug 14, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 apache#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 apache#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 apache#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
dataroaring
pushed a commit
that referenced
this issue
Aug 16, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 #5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 #6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp #7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 #8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 #9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 #10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 #11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 #12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 #13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 #14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 #15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 #16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 #17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 #18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 #19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 #20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 #21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 #22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 #23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h #24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 #25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 #26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 #27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 #28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 #29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 #30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 #31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 #32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 #33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 #34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 #38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 #39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 #40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
HappenLee
pushed a commit
that referenced
this issue
Sep 16, 2024
1. Add a new class MemCounter. 2. MemTracker and MemTrackerLimiter no longer have inheritance and dependencies. 3. MemTrackerLimiter is used to count process memory, the BE web page `/mem_tracker` also only displays MemTrackerLimiter. 4. MemTracker is used to count the memory of operators and some data structures. It is not counted in the process memory and is used for logic control and profile. In addition, it seems that the crash is caused by memory abnormality, not sure whether this PR can fix the problem, but this will help locate the problem and more elegant. ``` ==6641==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000980e70 at pc 0x559be21880be bp 0x7fd7cfc75070 sp 0x7fd7cfc75068 READ of size 8 at 0x603000980e70 thread T1453 (memory_maintena) #0 0x559be21880bd in std::__atomic_base<long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 #1 0x559be21880bd in doris::MemTracker::MemCounter::current_value() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:139:63 #2 0x559be21880bd in doris::MemTracker::consumption() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:165:56 #3 0x559be3985133 in doris::MemTrackerLimiter::refresh_global_counter() /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker_limiter.cpp:245:59 #4 0x559bdfcd52d5 in doris::Daemon::memory_maintenance_thread() /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:239:13 #5 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 #6 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 #7 0x7fdf5647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x603000980e74 is located 0 bytes after 20-byte region [0x603000980e60,0x603000980e74) 04:19:48 freed by thread T1491 (Scan_normal [wo) here: 04:19:48 #0 0x559bdfb0dd9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x33546d9d) (BuildId: f170e92ad3c55512) 04:19:48 #1 0x559bdfb1d42b in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2 04:19:48 #2 0x559bdfb1d42b in std::allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27 04:19:48 #3 0x559bdfb1d42b in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13 04:19:48 #4 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:237:9 04:19:48 #5 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:232:4 04:19:48 #6 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658:9 04:19:48 #7 0x559bdfb22501 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 04:19:48 #8 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 04:19:48 #9 0x559bdfb22501 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 04:19:48 #10 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 04:19:48 #11 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 04:19:48 #12 0x559bdfb22501 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 04:19:48 #13 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9867:1 04:19:48 #14 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9866:36 04:19:48 #15 0x559be0047ee6 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:372:5 04:19:48 #16 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13 04:19:48 #17 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28 04:19:48 #18 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18 04:19:48 #19 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18 04:19:48 #20 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5 04:19:48 #21 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32 04:19:48 #22 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 04:19:48 #23 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 04:19:48 #24 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 04:19:48 #25 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 04:19:48 #26 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 04:19:48 #27 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 04:19:48 #28 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 04:19:48 #29 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 04:19:48 #30 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 04:19:48 04:19:48 previously allocated by thread T1491 (Scan_normal [wo) here: 04:19:48 #0 0x559bdfb0d53d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x3354653d) (BuildId: f170e92ad3c55512) 04:19:48 #1 0x559bdfb33f74 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14 04:19:48 #2 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct_aux<char*>(char*, char*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11 04:19:48 #3 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4 04:19:48 #4 0x559bdfb32705 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:451:9 04:19:48 #5 0x559bdfc326e6 in decltype(::new((void*)(0)) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>(std::declval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>())) std::construct_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39 04:19:48 #6 0x559bdfc326e6 in void std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4 04:19:48 #7 0x559bdfc326e6 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4 04:19:48 #8 0x559be0032206 in doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:300:33 04:19:48 #9 0x559be0047aa3 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:355:9 04:19:48 #10 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13 04:19:48 #11 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28 04:19:48 #12 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18 04:19:48 #13 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18 04:19:48 #14 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5 04:19:48 #15 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32 04:19:48 #16 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 04:19:48 #17 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 04:19:48 #18 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 04:19:48 #19 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 04:19:48 #20 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 04:19:48 #21 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 04:19:48 #22 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 04:19:48 #23 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 04:19:48 #24 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 ```
mrhhsg
added a commit
to mrhhsg/doris
that referenced
this issue
Sep 21, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 apache#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 apache#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 apache#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 apache#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
xinyiZzz
added a commit
to xinyiZzz/incubator-doris
that referenced
this issue
Oct 16, 2024
1. Add a new class MemCounter. 2. MemTracker and MemTrackerLimiter no longer have inheritance and dependencies. 3. MemTrackerLimiter is used to count process memory, the BE web page `/mem_tracker` also only displays MemTrackerLimiter. 4. MemTracker is used to count the memory of operators and some data structures. It is not counted in the process memory and is used for logic control and profile. In addition, it seems that the crash is caused by memory abnormality, not sure whether this PR can fix the problem, but this will help locate the problem and more elegant. ``` ==6641==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000980e70 at pc 0x559be21880be bp 0x7fd7cfc75070 sp 0x7fd7cfc75068 READ of size 8 at 0x603000980e70 thread T1453 (memory_maintena) #0 0x559be21880bd in std::__atomic_base<long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 #1 0x559be21880bd in doris::MemTracker::MemCounter::current_value() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:139:63 apache#2 0x559be21880bd in doris::MemTracker::consumption() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:165:56 apache#3 0x559be3985133 in doris::MemTrackerLimiter::refresh_global_counter() /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker_limiter.cpp:245:59 apache#4 0x559bdfcd52d5 in doris::Daemon::memory_maintenance_thread() /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:239:13 apache#5 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#6 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#7 0x7fdf5647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x603000980e74 is located 0 bytes after 20-byte region [0x603000980e60,0x603000980e74) 04:19:48 freed by thread T1491 (Scan_normal [wo) here: 04:19:48 #0 0x559bdfb0dd9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x33546d9d) (BuildId: f170e92ad3c55512) 04:19:48 #1 0x559bdfb1d42b in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2 04:19:48 apache#2 0x559bdfb1d42b in std::allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27 04:19:48 apache#3 0x559bdfb1d42b in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13 04:19:48 apache#4 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:237:9 04:19:48 apache#5 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:232:4 04:19:48 apache#6 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658:9 04:19:48 apache#7 0x559bdfb22501 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 04:19:48 apache#8 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 04:19:48 apache#9 0x559bdfb22501 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 04:19:48 apache#10 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 04:19:48 apache#11 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 04:19:48 apache#12 0x559bdfb22501 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 04:19:48 apache#13 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9867:1 04:19:48 apache#14 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9866:36 04:19:48 apache#15 0x559be0047ee6 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:372:5 04:19:48 apache#16 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13 04:19:48 apache#17 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28 04:19:48 apache#18 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18 04:19:48 apache#19 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18 04:19:48 apache#20 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5 04:19:48 apache#21 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32 04:19:48 apache#22 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 04:19:48 apache#23 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 04:19:48 apache#24 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 04:19:48 apache#25 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 04:19:48 apache#26 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 04:19:48 apache#27 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 04:19:48 apache#28 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 04:19:48 apache#29 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 04:19:48 apache#30 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 04:19:48 04:19:48 previously allocated by thread T1491 (Scan_normal [wo) here: 04:19:48 #0 0x559bdfb0d53d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x3354653d) (BuildId: f170e92ad3c55512) 04:19:48 #1 0x559bdfb33f74 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14 04:19:48 apache#2 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct_aux<char*>(char*, char*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11 04:19:48 apache#3 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4 04:19:48 apache#4 0x559bdfb32705 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:451:9 04:19:48 apache#5 0x559bdfc326e6 in decltype(::new((void*)(0)) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>(std::declval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>())) std::construct_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39 04:19:48 apache#6 0x559bdfc326e6 in void std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4 04:19:48 apache#7 0x559bdfc326e6 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4 04:19:48 apache#8 0x559be0032206 in doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:300:33 04:19:48 apache#9 0x559be0047aa3 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:355:9 04:19:48 apache#10 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13 04:19:48 apache#11 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28 04:19:48 apache#12 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18 04:19:48 apache#13 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18 04:19:48 apache#14 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5 04:19:48 apache#15 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32 04:19:48 apache#16 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 04:19:48 apache#17 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 04:19:48 apache#18 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 04:19:48 apache#19 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 04:19:48 apache#20 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 04:19:48 apache#21 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 04:19:48 apache#22 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 04:19:48 apache#23 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 04:19:48 apache#24 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8 ```
airborne12
added a commit
that referenced
this issue
Oct 22, 2024
…dir when index file writer open index (#42207) ## Proposed changes Fix UAF as below ``` ==16442==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0008718a0 at pc 0x5586663cea7a bp 0x7f02673d0530 sp 0x7f02673d0528 READ of size 8 at 0x60f0008718a0 thread T490 (CumuCompactionT) #0 0x5586663cea79 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::finish() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:549:63 #1 0x5586663360bc in doris::segment_v2::ScalarColumnWriter::write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:640:41 #2 0x5586662ce160 in doris::segment_v2::SegmentWriter::_write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1435:9 #3 0x5586662ccb13 in doris::segment_v2::SegmentWriter::finalize_columns_index(unsigned long*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1317:5 #4 0x55866653c074 in doris::VerticalBetaRowsetWriter::_flush_columns(doris::segment_v2::SegmentWriter*, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:127:5 #5 0x558666534e3f in doris::VerticalBetaRowsetWriter::flush_columns(bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:153:5 #6 0x558665a8d8b2 in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:333:5 #7 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 #8 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 #9 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 #10 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 #11 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 #12 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 #13 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #14 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #15 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #16 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 #17 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 #18 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 #19 0x7f0639e0d84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x60f0008718a0 is located 0 bytes inside of 168-byte region [0x60f0008718a0,0x60f000871948) freed by thread T490 (CumuCompactionT) here: #0 0x558663259d9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a1d9d) (BuildId: fa8094411569cb8d) #1 0x558665a655eb in std::default_delete::operator()(lucene::store::Directory*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x558665a655eb in std::unique_ptr>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x558665f83dfe in std::pair, std::allocator>> const, std::unique_ptr>>::~pair() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:2379:12 #4 0x558665f83dfe in void std::destroy_at, std::allocator>> const, std::unique_ptr>>>(std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 #5 0x558665f83dfe in void std::allocator_traits, std::allocator>> const, std::unique_ptr>>>>>::destroy, std::allocator>> const, std::unique_ptr>>>(std::allocator, std::allocator>> const, std::unique_ptr>>>>&, std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4 #6 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_destroy_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:623:2 #7 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_drop_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:631:2 #8 0x558665f9e9b8 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_emplace_unique, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:2391:6 #9 0x558665f86e29 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::map, std::allocator>>, std::unique_ptr>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::emplace, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:577:16 #10 0x558665f86e29 in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:60:19 #11 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 #12 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 #13 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 #14 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 #15 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 #16 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 #17 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 #18 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 #19 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 #20 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 #21 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 #22 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 #23 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 #24 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 #25 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 #26 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 #27 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #28 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #29 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #30 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 #31 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 #32 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 previously allocated by thread T490 (CumuCompactionT) here: #0 0x55866325953d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a153d) (BuildId: fa8094411569cb8d) #1 0x558665fb7d7b in doris::segment_v2::DorisFSDirectoryFactory::getDirectory(std::shared_ptr const&, char const*, bool, lucene::store::LockFactory*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp:797:15 #2 0x558665f86b5c in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:58:17 #3 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 #4 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 #5 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 #6 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 #7 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 #8 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 #9 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 #10 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 #11 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 #12 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 #13 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 #14 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 #15 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 #16 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 #17 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 #18 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 #19 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #20 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #21 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #22 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 #23 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 #24 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 ```
airborne12
added a commit
to airborne12/apache-doris
that referenced
this issue
Oct 23, 2024
…dir when index file writer open index (apache#42207) Fix UAF as below ``` ==16442==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0008718a0 at pc 0x5586663cea7a bp 0x7f02673d0530 sp 0x7f02673d0528 READ of size 8 at 0x60f0008718a0 thread T490 (CumuCompactionT) #0 0x5586663cea79 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::finish() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:549:63 #1 0x5586663360bc in doris::segment_v2::ScalarColumnWriter::write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:640:41 #2 0x5586662ce160 in doris::segment_v2::SegmentWriter::_write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1435:9 apache#3 0x5586662ccb13 in doris::segment_v2::SegmentWriter::finalize_columns_index(unsigned long*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1317:5 apache#4 0x55866653c074 in doris::VerticalBetaRowsetWriter::_flush_columns(doris::segment_v2::SegmentWriter*, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:127:5 apache#5 0x558666534e3f in doris::VerticalBetaRowsetWriter::flush_columns(bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:153:5 apache#6 0x558665a8d8b2 in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:333:5 apache#7 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#8 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#9 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#10 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#11 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#12 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#13 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#14 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#15 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#16 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#17 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#18 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 apache#19 0x7f0639e0d84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x60f0008718a0 is located 0 bytes inside of 168-byte region [0x60f0008718a0,0x60f000871948) freed by thread T490 (CumuCompactionT) here: #0 0x558663259d9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a1d9d) (BuildId: fa8094411569cb8d) #1 0x558665a655eb in std::default_delete::operator()(lucene::store::Directory*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x558665a655eb in std::unique_ptr>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 apache#3 0x558665f83dfe in std::pair, std::allocator>> const, std::unique_ptr>>::~pair() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:2379:12 apache#4 0x558665f83dfe in void std::destroy_at, std::allocator>> const, std::unique_ptr>>>(std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#5 0x558665f83dfe in void std::allocator_traits, std::allocator>> const, std::unique_ptr>>>>>::destroy, std::allocator>> const, std::unique_ptr>>>(std::allocator, std::allocator>> const, std::unique_ptr>>>>&, std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4 apache#6 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_destroy_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:623:2 apache#7 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_drop_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:631:2 apache#8 0x558665f9e9b8 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_emplace_unique, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:2391:6 apache#9 0x558665f86e29 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::map, std::allocator>>, std::unique_ptr>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::emplace, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:577:16 apache#10 0x558665f86e29 in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:60:19 apache#11 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 apache#12 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 apache#13 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 apache#14 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 apache#15 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 apache#16 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 apache#17 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 apache#18 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 apache#19 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 apache#20 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 apache#21 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#22 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#23 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#24 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#25 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#26 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#27 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#28 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#29 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#30 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#31 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#32 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 previously allocated by thread T490 (CumuCompactionT) here: #0 0x55866325953d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a153d) (BuildId: fa8094411569cb8d) #1 0x558665fb7d7b in doris::segment_v2::DorisFSDirectoryFactory::getDirectory(std::shared_ptr const&, char const*, bool, lucene::store::LockFactory*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp:797:15 #2 0x558665f86b5c in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:58:17 apache#3 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 apache#4 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 apache#5 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 apache#6 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 apache#7 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 apache#8 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 apache#9 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 apache#10 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 apache#11 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 apache#12 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 apache#13 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#14 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#15 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#16 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#17 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#18 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#19 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#20 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#21 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#22 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#23 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#24 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
be/build/CMakeFiles/CMakeError.log:
The text was updated successfully, but these errors were encountered: