Skip to content

branch-3.1: [fix](load)fix load json format boolean type to int column. #54397 #54641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Aug 14, 2025
Merged

branch-3.1: [fix](load)fix load json format boolean type to int column. #54397 #54641

morrySnow merged 1 commit into from
Aug 14, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 12, 2025

Cherry-picked from #54397

@hubgeter
[fix](load)fix load json format boolean type to int column. (#54397)
e4d79b8 
### What problem does this PR solve?

Related PR: #43469
Problem Summary:
PR #43469 accidentally removed the logic for reading boolean
"true"/"false" values in the simd join reader. Before PR #43469,
"true"/"false" were treated as "1"/"0", allowing a bool column in a JSON
file to be imported into an Doris int column. This PR restores this
logic.
@github-actions github-actions bot requested a review from morrySnow as a code owner August 12, 2025 22:53
@Thearas
Copy link
Contributor

Thearas commented Aug 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 12, 2025
@Thearas
Copy link
Contributor

Thearas commented Aug 12, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 12, 2025

TPC-H: Total hot run time: 32626 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4d79b83335e739324b7c9e227daacca46b2a4cf, data reload: false

------ Round 1 ----------------------------------
q1	17584	5506	5385	5385
q2	2024	416	280	280
q3	11873	1226	744	744
q4	10537	869	447	447
q5	9439	2346	2128	2128
q6	178	163	131	131
q7	906	744	619	619
q8	9328	1450	1131	1131
q9	5174	4924	4893	4893
q10	6751	2273	1841	1841
q11	484	273	261	261
q12	338	354	209	209
q13	17778	3574	2961	2961
q14	229	230	211	211
q15	545	460	482	460
q16	421	427	384	384
q17	586	860	355	355
q18	6824	6738	6517	6517
q19	1196	951	522	522
q20	314	332	200	200
q21	2729	2169	1965	1965
q22	1044	1002	982	982
Total cold run time: 106282 ms
Total hot run time: 32626 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5485	5457	5478	5457
q2	234	324	243	243
q3	2219	2623	2309	2309
q4	1353	1801	1362	1362
q5	4335	4915	4898	4898
q6	166	163	129	129
q7	2046	1954	1800	1800
q8	2588	2802	2681	2681
q9	7201	7191	7205	7191
q10	3048	3321	2763	2763
q11	561	496	499	496
q12	652	800	610	610
q13	3378	3768	3135	3135
q14	284	291	280	280
q15	517	488	465	465
q16	436	475	425	425
q17	1204	1705	1250	1250
q18	7711	7359	7478	7359
q19	757	924	1121	924
q20	2034	2025	1900	1900
q21	5357	4955	4615	4615
q22	1092	1059	993	993
Total cold run time: 52658 ms
Total hot run time: 51285 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2025

TPC-DS: Total hot run time: 192495 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4d79b83335e739324b7c9e227daacca46b2a4cf, data reload: false

query1	947	402	399	399
query2	6194	1985	1886	1886
query3	8692	196	194	194
query4	33376	24433	23531	23531
query5	3941	593	446	446
query6	290	202	199	199
query7	4203	505	311	311
query8	290	242	235	235
query9	9383	2553	2518	2518
query10	477	322	264	264
query11	18116	15444	15140	15140
query12	154	107	107	107
query13	1559	552	433	433
query14	9240	7581	7372	7372
query15	206	200	181	181
query16	8053	649	473	473
query17	1594	792	600	600
query18	2166	417	345	345
query19	237	201	171	171
query20	136	128	121	121
query21	208	129	110	110
query22	4590	4589	4547	4547
query23	35240	33939	34538	33939
query24	7747	2679	2689	2679
query25	517	505	445	445
query26	780	282	174	174
query27	2089	489	371	371
query28	5466	2183	2151	2151
query29	637	608	470	470
query30	255	190	158	158
query31	1028	907	828	828
query32	70	62	60	60
query33	509	398	308	308
query34	745	859	509	509
query35	797	811	737	737
query36	1001	1072	946	946
query37	101	91	72	72
query38	4013	3989	3967	3967
query39	1541	1491	1461	1461
query40	203	123	106	106
query41	51	49	49	49
query42	122	109	105	105
query43	505	528	475	475
query44	1341	810	804	804
query45	187	180	169	169
query46	885	1061	680	680
query47	1964	1979	1934	1934
query48	403	428	341	341
query49	723	494	442	442
query50	671	710	431	431
query51	7272	7183	7237	7183
query52	100	96	91	91
query53	224	262	187	187
query54	553	543	489	489
query55	85	79	81	79
query56	293	272	270	270
query57	1267	1237	1175	1175
query58	247	224	213	213
query59	3101	3304	3089	3089
query60	307	296	270	270
query61	110	112	109	109
query62	761	759	709	709
query63	231	195	196	195
query64	3466	1000	678	678
query65	3405	3323	3302	3302
query66	810	414	306	306
query67	16279	15863	15576	15576
query68	7426	818	532	532
query69	491	302	267	267
query70	1182	1135	1148	1135
query71	406	305	257	257
query72	5740	3636	3827	3636
query73	624	739	344	344
query74	10346	9441	9279	9279
query75	3273	3197	2662	2662
query76	3114	1173	777	777
query77	662	347	283	283
query78	10385	10491	9537	9537
query79	2950	882	591	591
query80	637	528	432	432
query81	501	263	221	221
query82	573	124	96	96
query83	187	170	147	147
query84	284	101	76	76
query85	773	359	299	299
query86	347	307	305	305
query87	4344	4347	4226	4226
query88	4806	2378	2368	2368
query89	402	327	292	292
query90	1836	186	187	186
query91	131	141	106	106
query92	68	55	51	51
query93	1392	863	529	529
query94	690	369	310	310
query95	338	277	263	263
query96	473	598	286	286
query97	3217	3299	3173	3173
query98	230	207	202	202
query99	1538	1418	1312	1312
Total cold run time: 291520 ms
Total hot run time: 192495 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2025

ClickBench: Total hot run time: 29.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4d79b83335e739324b7c9e227daacca46b2a4cf, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.03	0.03
query3	0.24	0.06	0.06
query4	1.63	0.11	0.10
query5	0.53	0.52	0.52
query6	1.16	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.52	0.50
query10	0.57	0.56	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	0.79	0.82	0.78
query15	0.83	0.83	0.81
query16	0.38	0.39	0.39
query17	1.06	1.02	1.08
query18	0.23	0.21	0.23
query19	1.99	1.88	1.78
query20	0.02	0.02	0.01
query21	15.45	0.88	0.59
query22	0.73	0.72	0.60
query23	15.30	1.45	0.57
query24	3.26	1.49	2.17
query25	0.13	0.21	0.09
query26	0.18	0.15	0.14
query27	0.06	0.06	0.04
query28	14.87	0.94	0.42
query29	12.57	3.91	3.21
query30	0.24	0.09	0.07
query31	2.85	0.58	0.37
query32	3.23	0.53	0.45
query33	2.96	3.00	3.02
query34	16.69	5.20	4.52
query35	4.62	4.52	4.59
query36	0.63	0.49	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 105.33 s
Total hot run time: 29.24 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.45% (12668/27874)
Line Coverage 36.31% (113014/311267)
Region Coverage 35.36% (58432/165268)
Branch Coverage 32.58% (31821/97662)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.54% (21341/27524)
Line Coverage 71.76% (223058/310838)
Region Coverage 69.73% (133536/191504)
Branch Coverage 63.30% (69709/110120)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.54% (21341/27524)
Line Coverage 71.76% (223058/310838)
Region Coverage 69.73% (133536/191504)
Branch Coverage 63.30% (69709/110120)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.47% (21324/27524)
Line Coverage 71.73% (222956/310838)
Region Coverage 69.71% (133507/191504)
Branch Coverage 63.28% (69687/110120)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.24% (20985/27524)
Line Coverage 69.62% (216419/310838)
Region Coverage 67.63% (129515/191504)
Branch Coverage 61.18% (67376/110120)

@morrySnow morrySnow merged commit a07ab32 into branch-3.1 Aug 14, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-54397-branch-3.1 branch August 14, 2025 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @hubgeter