[Fix](Variant) fix variant cast to jsonb into wrong NULL values #50180

eldenmoon · 2025-04-18T11:24:11Z

implement cast to jsonb in ConvertImplGenericFromVariant
implement DataTypeObjectSerDe::serialize_column_to_json

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

1. implement cast to jsonb in `ConvertImplGenericFromVariant` 2. implement `DataTypeObjectSerDe::serialize_column_to_json`

Thearas · 2025-04-18T11:24:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2025-04-18T11:24:20Z

run buildall

doris-robot · 2025-04-18T12:11:55Z

TPC-H: Total hot run time: 34254 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e1c77a621a23fbd455b8e0b8343a672ed22dcb8, data reload: false

------ Round 1 ----------------------------------
q1	25856	5092	5025	5025
q2	2061	283	178	178
q3	10391	1250	719	719
q4	10236	996	565	565
q5	7551	2404	2382	2382
q6	181	172	131	131
q7	915	749	621	621
q8	9316	1357	1205	1205
q9	6901	5139	5140	5139
q10	6845	2323	1882	1882
q11	512	290	294	290
q12	367	358	225	225
q13	17787	3686	3090	3090
q14	237	226	208	208
q15	531	487	481	481
q16	444	443	408	408
q17	595	866	374	374
q18	7495	7241	7034	7034
q19	1639	960	570	570
q20	343	340	227	227
q21	4285	3433	2497	2497
q22	1043	1019	1003	1003
Total cold run time: 115531 ms
Total hot run time: 34254 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5170	5096	5126	5096
q2	240	320	235	235
q3	2181	2633	2281	2281
q4	1430	1809	1419	1419
q5	4468	4457	4423	4423
q6	218	165	129	129
q7	1980	1897	1793	1793
q8	2621	2526	2594	2526
q9	7221	7251	7070	7070
q10	2990	3191	2725	2725
q11	579	512	509	509
q12	699	779	614	614
q13	3555	3928	3278	3278
q14	298	315	278	278
q15	514	485	484	484
q16	471	502	467	467
q17	1169	1599	1411	1411
q18	7708	7660	7383	7383
q19	834	896	1004	896
q20	1997	1953	1828	1828
q21	5307	4931	4714	4714
q22	1096	1073	1038	1038
Total cold run time: 52746 ms
Total hot run time: 50597 ms

doris-robot · 2025-04-18T12:24:04Z

TPC-DS: Total hot run time: 191972 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e1c77a621a23fbd455b8e0b8343a672ed22dcb8, data reload: false

query1	1378	1110	1091	1091
query2	6175	1812	1795	1795
query3	11102	4606	4594	4594
query4	53608	24273	23057	23057
query5	4883	583	454	454
query6	338	204	214	204
query7	4872	494	275	275
query8	315	265	243	243
query9	5197	2574	2571	2571
query10	408	321	279	279
query11	14961	14936	14791	14791
query12	149	110	104	104
query13	1012	517	381	381
query14	9992	6331	6598	6331
query15	198	190	171	171
query16	7062	694	481	481
query17	1106	757	611	611
query18	1565	419	323	323
query19	200	238	177	177
query20	136	131	123	123
query21	203	120	105	105
query22	4440	4471	4305	4305
query23	34082	33392	33517	33392
query24	6582	2448	2418	2418
query25	447	467	399	399
query26	724	267	152	152
query27	2421	514	347	347
query28	3067	2149	2108	2108
query29	576	564	431	431
query30	268	211	182	182
query31	867	856	773	773
query32	72	61	65	61
query33	444	360	308	308
query34	764	863	509	509
query35	803	862	764	764
query36	984	1000	926	926
query37	112	99	81	81
query38	4144	4343	4112	4112
query39	1484	1472	1447	1447
query40	220	121	109	109
query41	60	51	53	51
query42	124	115	107	107
query43	494	510	481	481
query44	1321	824	844	824
query45	187	172	170	170
query46	846	1028	635	635
query47	1827	1880	1792	1792
query48	381	405	305	305
query49	684	519	429	429
query50	680	701	432	432
query51	4265	4234	4307	4234
query52	120	108	100	100
query53	228	260	182	182
query54	586	572	520	520
query55	83	81	86	81
query56	313	311	310	310
query57	1187	1189	1117	1117
query58	255	263	246	246
query59	2721	2765	2675	2675
query60	346	326	309	309
query61	128	128	130	128
query62	797	746	674	674
query63	233	192	208	192
query64	1869	1096	715	715
query65	4425	4289	4230	4230
query66	748	400	306	306
query67	15944	15678	15255	15255
query68	7342	887	508	508
query69	527	305	262	262
query70	1164	1112	1057	1057
query71	507	307	285	285
query72	5915	4798	4648	4648
query73	1534	580	347	347
query74	8837	9100	8734	8734
query75	4034	3183	2706	2706
query76	4198	1191	755	755
query77	738	367	282	282
query78	10049	10229	9348	9348
query79	2027	776	549	549
query80	581	510	453	453
query81	481	254	216	216
query82	455	124	97	97
query83	252	254	269	254
query84	293	97	87	87
query85	786	349	320	320
query86	342	298	297	297
query87	4313	4405	4386	4386
query88	3589	2211	2181	2181
query89	394	313	289	289
query90	1949	207	211	207
query91	142	148	113	113
query92	71	58	60	58
query93	1253	949	586	586
query94	667	423	306	306
query95	372	295	293	293
query96	471	551	273	273
query97	3132	3202	3100	3100
query98	226	212	201	201
query99	1450	1420	1289	1289
Total cold run time: 296304 ms
Total hot run time: 191972 ms

doris-robot · 2025-04-18T12:29:18Z

ClickBench: Total hot run time: 29.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e1c77a621a23fbd455b8e0b8343a672ed22dcb8, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.20
query4	1.60	0.20	0.19
query5	0.58	0.59	0.59
query6	1.22	0.70	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.52	0.51
query10	0.56	0.56	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.61
query14	1.15	1.17	1.17
query15	0.88	0.86	0.85
query16	0.39	0.37	0.37
query17	1.01	1.01	1.03
query18	0.21	0.20	0.20
query19	1.92	1.78	1.84
query20	0.01	0.01	0.02
query21	15.41	0.92	0.56
query22	0.74	1.11	0.66
query23	15.05	1.40	0.63
query24	6.73	1.91	0.58
query25	0.52	0.20	0.06
query26	0.65	0.16	0.14
query27	0.05	0.06	0.05
query28	10.09	0.89	0.44
query29	12.55	4.09	3.40
query30	0.25	0.09	0.06
query31	2.82	0.62	0.38
query32	3.22	0.54	0.47
query33	3.00	3.01	3.14
query34	15.62	5.09	4.52
query35	4.53	4.51	4.59
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.18	0.13	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.89 s
Total hot run time: 29.41 s

doris-robot · 2025-04-18T13:21:21Z

BE UT Coverage Report

Increment line coverage 0.00% (0/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.16% (14426/27137)
Line Coverage	42.02% (124998/297473)
Region Coverage	40.83% (63854/156392)
Branch Coverage	35.46% (32103/90540)

amorynan · 2025-04-21T02:54:45Z

be/src/vec/functions/function_cast.h

-                } else if (WhichDataType(data_type_to).is_string()) {
-                    return ConvertImplGenericToString::execute2(context, block, arguments, result,
-                                                                input_rows_count);
                } else {


Before this pr we cast(v as jsonb) whether to enter this branch, call Nullable(ColumnObject)-> insert_many_defaults() which cause the wrong NULL value ?

csun5285

LGTM

github-actions · 2025-04-21T03:30:41Z

PR approved by anyone and no changes requested.

airborne12

LGTM

github-actions · 2025-04-21T07:34:21Z

PR approved by at least one committer and no changes requested.

…he#50180) 1. implement cast to jsonb in `ConvertImplGenericFromVariant` 2. implement `DataTypeObjectSerDe::serialize_column_to_json`

…wrong NULL values (#50235) cherry-pick from #50180

…wrong NULL values (#50180) (#50234) cherry-pick from #50180

…he#50180) 1. implement cast to jsonb in `ConvertImplGenericFromVariant` 2. implement `DataTypeObjectSerDe::serialize_column_to_json`

[Fix](Variant) fix variant cast to jsonb into wrong NULL values

9e1c77a

1. implement cast to jsonb in `ConvertImplGenericFromVariant` 2. implement `DataTypeObjectSerDe::serialize_column_to_json`

eldenmoon added dev/2.1.x dev/3.0.x labels Apr 18, 2025

amorynan reviewed Apr 21, 2025

View reviewed changes

csun5285 approved these changes Apr 21, 2025

View reviewed changes

github-actions bot added the reviewed label Apr 21, 2025

airborne12 approved these changes Apr 21, 2025

View reviewed changes

eldenmoon merged commit 4572b59 into apache:master Apr 21, 2025
29 of 33 checks passed

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2025

eldenmoon added usercase Important user case type label and removed approved Indicates a PR has been approved by one committer. labels Apr 21, 2025

eldenmoon deleted the fix-var-jsonb branch April 21, 2025 07:34

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Apr 21, 2025

eldenmoon mentioned this pull request Apr 21, 2025

Branch-3.0 Cherry-pick [Fix](Variant) fix variant cast to jsonb into wrong NULL values (#50180) #50234

Merged

eldenmoon mentioned this pull request Apr 21, 2025

Branch-2.1 Cherry-pick [Fix](Variant) fix variant cast to jsonb into wrong NULL values #50235

Merged

yiguolei pushed a commit that referenced this pull request Apr 22, 2025

Branch-2.1 Cherry-pick [Fix](Variant) fix variant cast to jsonb into …

9123d73

…wrong NULL values (#50235) cherry-pick from #50180

yiguolei added dev/2.1.10-merged and removed dev/2.1.x dev/2.1.x-conflict labels Apr 22, 2025

dataroaring pushed a commit that referenced this pull request Apr 22, 2025

Branch-3.0 Cherry-pick [Fix](Variant) fix variant cast to jsonb into …

415c2fc

…wrong NULL values (#50180) (#50234) cherry-pick from #50180

dataroaring added dev/3.0.6-merged and removed dev/3.0.x dev/3.0.x-conflict labels Apr 22, 2025

yiguolei mentioned this pull request May 13, 2025

2.1.10 Release Note #50853

Closed

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Closed

[Fix](Variant) fix variant cast to jsonb into wrong NULL values #50180

[Fix](Variant) fix variant cast to jsonb into wrong NULL values #50180

Uh oh!

Conversation

eldenmoon commented Apr 18, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 18, 2025

Uh oh!

eldenmoon commented Apr 18, 2025

Uh oh!

doris-robot commented Apr 18, 2025

Uh oh!

doris-robot commented Apr 18, 2025

Uh oh!

doris-robot commented Apr 18, 2025

Uh oh!

doris-robot commented Apr 18, 2025

BE UT Coverage Report

Uh oh!

amorynan Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

eldenmoon Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 21, 2025

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Apr 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants