Skip to content

[fix](load)fix load json format boolean type to int column. #54397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Aug 12, 2025
Merged

[fix](load)fix load json format boolean type to int column. #54397

morningman merged 2 commits into from
Aug 12, 2025
+52 −0

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Aug 6, 2025

What problem does this PR solve?

Related PR: #43469
Problem Summary:
PR #43469 accidentally removed the logic for reading boolean "true"/"false" values in the simd join reader. Before PR #43469, "true"/"false" were treated as "1"/"0", allowing a bool column in a JSON file to be imported into an Doris int column. This PR restores this logic.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Aug 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter hubgeter marked this pull request as ready for review August 6, 2025 08:39
@hubgeter
Copy link
Contributor Author

hubgeter commented Aug 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

TPC-H: Total hot run time: 33972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58d6a34b136beefd294a75eeaa462baaccd0979b, data reload: false

------ Round 1 ----------------------------------
q1	17594	5351	5280	5280
q2	1938	313	192	192
q3	10271	1434	724	724
q4	10212	1112	512	512
q5	7551	2355	2380	2355
q6	179	170	135	135
q7	907	776	618	618
q8	9293	1289	1140	1140
q9	6731	5045	5063	5045
q10	6915	2366	1948	1948
q11	452	286	271	271
q12	351	364	222	222
q13	17775	3690	2998	2998
q14	227	240	210	210
q15	554	467	453	453
q16	442	422	378	378
q17	583	827	362	362
q18	7309	7087	7016	7016
q19	4147	1095	581	581
q20	330	305	213	213
q21	3425	3070	2323	2323
q22	1055	1028	996	996
Total cold run time: 108241 ms
Total hot run time: 33972 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5516	5651	5379	5379
q2	236	321	217	217
q3	2078	2567	2189	2189
q4	1374	1711	1352	1352
q5	4393	4513	4315	4315
q6	225	195	134	134
q7	2270	1925	1789	1789
q8	2547	2658	2462	2462
q9	7334	7563	7338	7338
q10	3210	3320	2911	2911
q11	552	553	509	509
q12	1025	1088	618	618
q13	3583	3774	3325	3325
q14	278	296	270	270
q15	521	452	460	452
q16	440	502	467	467
q17	1169	1693	1357	1357
q18	7747	7667	7736	7667
q19	4609	868	902	868
q20	2800	2075	1832	1832
q21	10499	4299	4412	4299
q22	1109	1012	998	998
Total cold run time: 63515 ms
Total hot run time: 50748 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

TPC-DS: Total hot run time: 170281 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58d6a34b136beefd294a75eeaa462baaccd0979b, data reload: false

============================================
query1	989	387	415	387
query2	6586	1771	1671	1671
query3	6747	227	223	223
query4	27114	23987	23447	23447
query5	4400	656	502	502
query6	312	241	232	232
query7	4638	516	310	310
query8	283	236	219	219
query9	8570	2995	2942	2942
query10	473	341	293	293
query11	15777	14951	14784	14784
query12	186	137	133	133
query13	1653	562	421	421
query14	8516	5904	5909	5904
query15	211	183	165	165
query16	7135	660	446	446
query17	979	787	658	658
query18	2034	440	313	313
query19	216	206	180	180
query20	156	153	136	136
query21	226	127	109	109
query22	3965	4095	3844	3844
query23	34653	34082	34348	34082
query24	5053	2423	2458	2423
query25	506	521	437	437
query26	711	294	164	164
query27	2259	521	363	363
query28	3026	2345	2325	2325
query29	616	608	507	507
query30	300	238	194	194
query31	839	819	705	705
query32	92	77	81	77
query33	475	477	368	368
query34	809	850	531	531
query35	815	857	755	755
query36	1006	1074	943	943
query37	130	116	97	97
query38	3972	3932	3961	3932
query39	1427	1395	1374	1374
query40	237	145	136	136
query41	64	59	54	54
query42	143	123	131	123
query43	527	515	496	496
query44	1462	897	882	882
query45	197	185	186	185
query46	944	1066	691	691
query47	1767	1812	1720	1720
query48	389	450	312	312
query49	695	516	456	456
query50	651	691	423	423
query51	4174	4178	4270	4178
query52	130	138	131	131
query53	265	301	226	226
query54	657	651	565	565
query55	97	91	90	90
query56	354	351	358	351
query57	1184	1222	1129	1129
query58	331	339	331	331
query59	2562	2639	2565	2565
query60	408	390	386	386
query61	122	137	123	123
query62	762	723	678	678
query63	251	222	209	209
query64	2335	1185	765	765
query65	4246	4176	4114	4114
query66	1057	448	329	329
query67	query68	18388	886	853	853
query69	1196	294	311	294
query70	1513	1039	1106	1039
query71	704	320	316	316
query72	9189	2255	2185	2185
query73	3750	702	359	359
query74	9092	8942	8865	8865
query75	7599	3088	2661	2661
query76	8919	1221	776	776
query77	1142	423	343	343
query78	9610	10033	9147	9147
query79	13661	619	572	572
query80	1609	550	544	544
query81	578	253	223	223
query82	338	159	117	117
query83	345	290	269	269
query84	303	96	78	78
query85	841	372	370	370
query86	361	336	330	330
query87	4184	4206	4068	4068
query88	3578	2250	2234	2234
query89	472	356	314	314
query90	2683	233	227	227
query91	140	139	110	110
query92	88	73	68	68
query93	4342	971	657	657
query94	905	400	271	271
query95	403	330	319	319
query96	508	597	283	283
query97	2716	2742	2616	2616
query98	276	233	225	225
query99	1459	1428	1304	1304
Total cold run time: 294758 ms
Total hot run time: 170281 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

ClickBench: Total hot run time: 33.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58d6a34b136beefd294a75eeaa462baaccd0979b, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.64	0.11	0.11
query5	0.41	0.42	0.42
query6	1.17	0.64	0.64
query7	0.03	0.01	0.02
query8	0.04	0.04	0.04
query9	0.56	0.47	0.46
query10	0.53	0.52	0.55
query11	0.16	0.11	0.10
query12	0.15	0.11	0.11
query13	0.65	0.65	0.63
query14	0.94	1.06	1.05
query15	0.95	0.89	0.88
query16	0.40	0.40	0.38
query17	1.08	1.08	1.12
query18	0.22	0.20	0.20
query19	1.92	1.79	1.83
query20	0.02	0.01	0.01
query21	15.37	0.85	0.57
query22	0.76	1.24	0.68
query23	14.90	1.18	0.64
query24	6.93	0.92	0.91
query25	0.52	0.11	0.15
query26	0.54	0.15	0.12
query27	0.06	0.05	0.06
query28	9.61	0.85	0.43
query29	12.58	3.92	3.42
query30	3.07	2.96	2.95
query31	2.83	0.55	0.38
query32	3.24	0.57	0.50
query33	3.09	3.12	3.21
query34	15.86	5.28	4.97
query35	4.87	4.97	5.00
query36	0.71	0.51	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.05
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 106.72 s
Total hot run time: 33.27 s

@doris-robot
Copy link

doris-robot commented Aug 6, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.31% (16384/28100)
Line Coverage 47.19% (148056/313727)
Region Coverage 36.09% (110637/306521)
Branch Coverage 39.02% (49179/126032)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 6, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.27% (22412/27577)
Line Coverage 73.97% (232003/313665)
Region Coverage 61.42% (193307/314724)
Branch Coverage 65.33% (83329/127560)

@morningman morningman added usercase Important user case type label dev/3.0.x dev/3.1.x labels Aug 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 11, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 11, 2025

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Aug 12, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 675b5f1 into apache:master Aug 12, 2025
29 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Aug 12, 2025
@hubgeter
[fix](load)fix load json format boolean type to int column. (#54397)
ed92ce7 
### What problem does this PR solve?

Related PR: #43469
Problem Summary:
PR #43469 accidentally removed the logic for reading boolean
"true"/"false" values in the simd join reader. Before PR #43469,
"true"/"false" were treated as "1"/"0", allowing a bool column in a JSON
file to be imported into an Doris int column. This PR restores this
logic.
@github-actions github-actions bot mentioned this pull request Aug 12, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Aug 12, 2025
@hubgeter
[fix](load)fix load json format boolean type to int column. (#54397)
e4d79b8 
### What problem does this PR solve?

Related PR: #43469
Problem Summary:
PR #43469 accidentally removed the logic for reading boolean
"true"/"false" values in the simd join reader. Before PR #43469,
"true"/"false" were treated as "1"/"0", allowing a bool column in a JSON
file to be imported into an Doris int column. This PR restores this
logic.
@github-actions github-actions bot mentioned this pull request Aug 12, 2025
Merged
dataroaring pushed a commit that referenced this pull request Aug 14, 2025
@github-actions @hubgeter
branch-3.0: [fix](load)fix load json format boolean type to int column.
27f74f7 
#54397 (#54640)

Cherry-picked from #54397

Co-authored-by: daidai <changyuwei@selectdb.com>
morrySnow pushed a commit that referenced this pull request Aug 14, 2025
@github-actions @hubgeter
branch-3.1: [fix](load)fix load json format boolean type to int column.
a07ab32 
#54397 (#54641)

Cherry-picked from #54397

Co-authored-by: daidai <changyuwei@selectdb.com>
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.8-merged dev/3.1.0-merged p0_r reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@hubgeter @Thearas @doris-robot @hello-stephen @morningman @liaoxin01 @dataroaring @morrySnow