add .tar.gz suffix #53

cyongli · 2017-08-29T03:23:40Z

commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)

commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>

Co-authored-by: deardeng <565620795@qq.com> Squashed commit of the following: commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)

commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <platonekosama@gmail.com> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <565620795@qq.com> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <814383175@qq.com> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <814383175@qq.com> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <565620795@qq.com> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <814383175@qq.com> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <565620795@qq.com> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <565620795@qq.com> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <565620795@qq.com> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <1091517373@qq.com> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <814383175@qq.com> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <565620795@qq.com> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <565620795@qq.com> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <platonekosama@gmail.com> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <platonekosama@gmail.com> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <565620795@qq.com> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <565620795@qq.com> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <565620795@qq.com> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <814383175@qq.com> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <565620795@qq.com> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <814383175@qq.com> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <36853835+platoneko@users.noreply.github.com> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <platonekosama@gmail.com> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <zouxinyi02@gmail.com> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <pxl290@qq.com> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: deardeng <565620795@qq.com>

…is not enough to provide all the data for the query (#33800) When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query. this depends on #33362 such as materialized view def is as following: > CREATE MATERIALIZED VIEW mv_10086 > BUILD IMMEDIATE REFRESH AUTO ON MANUAL > partition by(l_shipdate) > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ('replication_num' = '1') > AS > select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total > from lineitem > left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate > group by > l_shipdate, > o_orderdate, > l_partkey, > l_suppkey; the materialized view data is as following: +------------+-------------+-----------+-----------+-----------+ | l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total | +------------+-------------+-----------+-----------+-----------+ | 2023-10-18 | 2023-10-18 | 2 | 3 | 109.20 | | 2023-10-17 | 2023-10-17 | 2 | 3 | 99.50 | | 2023-10-19 | 2023-10-19 | 2 | 3 | 99.50 | +------------+-------------+-----------+-----------+-----------+ when we insert data to partition `2023-10-17`, if we run query as following ``` select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total from lineitem left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate group by l_shipdate, o_orderdate, l_partkey, l_suppkey; ``` query rewrite by materialzied view will fail with message `Check partition query used validation fail` if we turn on the switch `SET enable_materialized_view_union_rewrite = true;` default true we run the query above again, it will success and will use union all materialized view and origin query to response the query correctly. the plan is as following: ``` | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | l_shipdate[#52] | | o_orderdate[#53] | | l_partkey[#54] | | l_suppkey[#55] | | sum_total[#56] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 11:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 11 | | UNPARTITIONED | | | | 10:VUNION(756) | | | | | |----9:VAGGREGATE (merge finalize)(753) | | | | output: sum(partial_sum(o_totalprice)[#46])[#51] | | | | group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | cardinality=2 | | | | distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | | | | 8:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_shipdate[#42] | | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 2 | | | | PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 08 | | HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | 7:VAGGREGATE (update serialize)(747) | | | STREAMING | | | output: partial_sum(o_totalprice[#41])[#46] | | | group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40] | | | cardinality=2 | | | distribute expr lists: l_shipdate[#37] | | | | | 6:VHASH JOIN(741) | | | join op: RIGHT OUTER JOIN(PARTITIONED)[] | | | equal join conjunct: (o_orderkey[#21] = l_orderkey[#5]) | | | equal join conjunct: (o_orderdate[#25] = l_shipdate[#15]) | | | runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) | | | cardinality=2 | | | vec output tuple id: 4 | | | output tuple id: 4 | | | vIntermediate tuple ids: 3 | | | hash output slot ids: 6 7 24 25 15 | | | final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31] | | | final project output tuple id: 4 | | | distribute expr lists: o_orderkey[#21], o_orderdate[#25] | | | distribute expr lists: l_orderkey[#5], l_shipdate[#15] | | | | | |----3:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_orderkey[#5] | | | | | 5:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 3 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 05 | | HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | 4:VOlapScanNode(722) | | TABLE: union_db.orders(orders), PREAGGREGATION: ON | | runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25] | | partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ... | | cardinality=3, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 4 | | | | PARTITION: HASH_PARTITIONED: l_orderkey[#5] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15] | | | | 2:VOlapScanNode(729) | | TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON | | PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18') | | partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227 | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 5 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | RANDOM | | | | 0:VOlapScanNode(718) | | TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON | | partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ... | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | MaterializedView | | MaterializedViewRewriteSuccessAndChose: | | Names: mv_10086 | | MaterializedViewRewriteSuccessButNotChose: | | | | MaterializedViewRewriteFail: | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ```

add .tar.gz suffix

0a08a91

morningman merged commit 7c278c9 into apache:master Aug 29, 2017

stdpain added a commit to stdpain/incubator-doris that referenced this pull request Jul 5, 2021

support some bitmap function bitmap_count bitmap_and/or/xor (apache#53)

84d0815

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jul 15, 2021

support some bitmap function bitmap_count bitmap_and/or/xor (apache#53)

e5f7d90

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Sep 7, 2021

support some bitmap function bitmap_count bitmap_and/or/xor (apache#53)

ca5e01f

starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Jul 26, 2023

moka if case (apache#53)

58ca80b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add .tar.gz suffix #53

add .tar.gz suffix #53

cyongli commented Aug 29, 2017

add .tar.gz suffix #53

add .tar.gz suffix #53

Conversation

cyongli commented Aug 29, 2017