Skip to content

branch-3.1: [feat](outfile) support compression type for csv format in outfile and export #55392 #55561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 4, 2025
Merged

branch-3.1: [feat](outfile) support compression type for csv format in outfile and export #55392 #55561

morrySnow merged 1 commit into from
Sep 4, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 2, 2025

Cherry-picked from #55392

@morningman
[feat](outfile) support compression type for csv format in outfile an…
6b64363 
…d export (#55392)

### What problem does this PR solve?

Outfile:
```
select * from tbl1
into outfile "file:///tmp/my_file_"
FORMAT AS CSV
PROPERTIES
(
    "compress_type" = "gzip"
);
```

Export:
```
EXPORT TABLE test_export_csv TO "file:///tmp/test_"
PROPERTIES(
    "label" = "l4",
    "format" = "csv",
    "column_separator"=",",
    "compress_type" = "gzip"
);
```

Now support "plain", "gz", "bz2", "snappyblock", "lz4block", "zstd"
@github-actions github-actions bot requested a review from morrySnow as a code owner September 2, 2025 02:47
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Sep 2, 2025
@dataroaring dataroaring reopened this Sep 2, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 2, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.20% (1214/1495)
Line Coverage 65.64% (21794/33200)
Region Coverage 67.21% (10950/16292)
Branch Coverage 56.85% (5781/10168)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 2, 2025

FE UT Coverage Report

Increment line coverage 68.18% (15/22) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Sep 2, 2025

TPC-H: Total hot run time: 33256 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b643637241c5fb1b61aa2357737ab9537b4abe6, data reload: false

------ Round 1 ----------------------------------
q1	17603	5702	5556	5556
q2	2033	386	285	285
q3	12225	1259	773	773
q4	10253	876	461	461
q5	8425	2424	2207	2207
q6	190	169	140	140
q7	958	776	623	623
q8	9358	1491	1215	1215
q9	5232	4935	4968	4935
q10	6771	2288	1862	1862
q11	482	279	265	265
q12	341	353	220	220
q13	17787	3596	3015	3015
q14	243	223	204	204
q15	516	467	473	467
q16	429	434	371	371
q17	605	888	369	369
q18	6991	6605	6376	6376
q19	1418	974	578	578
q20	346	354	207	207
q21	2920	2241	2128	2128
q22	1070	1029	999	999
Total cold run time: 106196 ms
Total hot run time: 33256 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5967	5562	5561	5561
q2	241	329	238	238
q3	2240	2675	2324	2324
q4	1359	1778	1351	1351
q5	4407	5128	5081	5081
q6	173	162	133	133
q7	2103	1970	1799	1799
q8	2675	2857	2731	2731
q9	7275	7258	7192	7192
q10	3091	3231	2747	2747
q11	574	518	492	492
q12	656	779	596	596
q13	3448	3802	3212	3212
q14	276	308	271	271
q15	516	469	464	464
q16	442	488	438	438
q17	1235	1730	1252	1252
q18	7766	7464	7478	7464
q19	839	1188	1073	1073
q20	2030	2046	1885	1885
q21	5408	4914	4567	4567
q22	1095	1080	1035	1035
Total cold run time: 53816 ms
Total hot run time: 51906 ms

@doris-robot
Copy link

doris-robot commented Sep 2, 2025

TPC-DS: Total hot run time: 192703 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b643637241c5fb1b61aa2357737ab9537b4abe6, data reload: false

query1	956	404	416	404
query2	6219	1988	1955	1955
query3	8704	206	201	201
query4	33529	23748	23313	23313
query5	3600	606	477	477
query6	288	205	185	185
query7	4211	492	319	319
query8	316	251	244	244
query9	9580	2635	2602	2602
query10	447	343	269	269
query11	18162	15387	15227	15227
query12	160	111	108	108
query13	1562	558	443	443
query14	9512	7268	6599	6599
query15	242	202	184	184
query16	8092	625	530	530
query17	1584	768	598	598
query18	2140	421	344	344
query19	255	200	177	177
query20	134	130	129	129
query21	214	135	111	111
query22	4628	4626	4301	4301
query23	35308	34385	34089	34089
query24	7270	2758	2707	2707
query25	500	484	405	405
query26	875	290	171	171
query27	2156	493	375	375
query28	5132	2263	2218	2218
query29	680	592	458	458
query30	239	193	168	168
query31	1005	919	858	858
query32	86	59	61	59
query33	523	382	319	319
query34	789	867	523	523
query35	802	809	769	769
query36	993	1079	956	956
query37	101	93	67	67
query38	4058	4037	4070	4037
query39	1546	1494	1494	1494
query40	221	135	110	110
query41	51	48	49	48
query42	134	107	108	107
query43	537	540	497	497
query44	1352	830	839	830
query45	188	181	176	176
query46	915	1068	699	699
query47	1975	2028	1967	1967
query48	415	438	363	363
query49	741	490	422	422
query50	719	708	462	462
query51	7410	7405	7247	7247
query52	102	102	95	95
query53	238	285	204	204
query54	570	576	498	498
query55	88	83	83	83
query56	287	307	271	271
query57	1301	1252	1209	1209
query58	240	228	228	228
query59	3181	3280	3075	3075
query60	300	293	273	273
query61	118	115	121	115
query62	821	778	714	714
query63	242	217	201	201
query64	3671	1113	744	744
query65	3392	3273	3305	3273
query66	856	444	331	331
query67	16141	15746	15768	15746
query68	5053	872	578	578
query69	515	322	284	284
query70	1205	1150	1147	1147
query71	395	312	283	283
query72	5988	3693	3903	3693
query73	653	759	364	364
query74	10539	9158	8865	8865
query75	3221	3131	2698	2698
query76	2988	1187	794	794
query77	501	379	281	281
query78	10334	10364	9694	9694
query79	3896	851	605	605
query80	816	534	449	449
query81	510	254	228	228
query82	596	121	88	88
query83	167	162	146	146
query84	249	105	82	82
query85	784	362	300	300
query86	396	289	302	289
query87	4306	4291	4259	4259
query88	5173	2619	2428	2428
query89	420	327	306	306
query90	1771	195	195	195
query91	139	137	113	113
query92	64	54	52	52
query93	2490	879	563	563
query94	690	416	304	304
query95	346	285	271	271
query96	487	628	291	291
query97	3202	3297	3168	3168
query98	234	218	198	198
query99	1520	1430	1306	1306
Total cold run time: 292452 ms
Total hot run time: 192703 ms

@doris-robot
Copy link

doris-robot commented Sep 2, 2025

ClickBench: Total hot run time: 29.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b643637241c5fb1b61aa2357737ab9537b4abe6, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.64	0.09	0.08
query5	0.52	0.50	0.54
query6	1.13	0.74	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.61	0.51	0.50
query10	0.53	0.56	0.55
query11	0.17	0.12	0.11
query12	0.16	0.13	0.13
query13	0.62	0.60	0.59
query14	0.80	0.78	0.80
query15	0.86	0.85	0.84
query16	0.37	0.38	0.39
query17	1.04	1.03	1.02
query18	0.19	0.20	0.20
query19	2.00	1.81	1.80
query20	0.02	0.01	0.02
query21	15.37	0.98	0.66
query22	0.76	0.75	0.70
query23	14.80	1.48	0.67
query24	2.19	0.38	0.24
query25	0.14	0.08	0.08
query26	0.29	0.19	0.18
query27	0.08	0.08	0.09
query28	13.45	1.27	0.57
query29	12.66	4.16	3.46
query30	0.26	0.09	0.06
query31	2.82	0.62	0.40
query32	3.23	0.58	0.48
query33	3.08	3.06	3.06
query34	16.78	5.25	4.60
query35	4.59	4.69	4.67
query36	0.62	0.48	0.49
query37	0.20	0.16	0.16
query38	0.16	0.16	0.15
query39	0.05	0.04	0.05
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.05	0.04
Total cold run time: 102.98 s
Total hot run time: 29.44 s

@doris-robot
Copy link

doris-robot commented Sep 2, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.49% (12725/27974)
Line Coverage 36.38% (113489/311950)
Region Coverage 34.00% (64946/191019)
Branch Coverage 31.05% (34100/109828)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 2, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.42% (21018/27504)
Line Coverage 69.78% (216944/310899)
Region Coverage 67.68% (129777/191757)
Branch Coverage 61.22% (67544/110322)

@morrySnow morrySnow merged commit b41fcda into branch-3.1 Sep 4, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-55392-branch-3.1 branch September 4, 2025 02:36
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @morningman