Skip to content

branch-3.1: [fix](paimon)Handle oversized CHAR/VARCHAR fields in Paimon to Doris type mapping #55051 #55451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
github-actions wants to merge 1 commit into from
Closed

branch-3.1: [fix](paimon)Handle oversized CHAR/VARCHAR fields in Paimon to Doris type mapping #55051 #55451

github-actions wants to merge 1 commit into from

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 28, 2025

Cherry-picked from #55051

[fix](paimon)Handle oversized CHAR/VARCHAR fields in Paimon to Doris …
20a7923 
…type mapping (#55051)

### What problem does this PR solve?
In PR #49623, we implemented
conversion from Paimon `VARCHAR/CHAR` types to Doris `VARCHAR/CHAR`
types. However, there are significant differences in the maximum length
constraints between these systems:

**Apache Paimon:**
- `CHAR` : Fixed-length character string declared using CHAR(n) where n
is the number of code points. n must have a value between `1` and
`2,147,483,647` (inclusive). Defaults to n=1 if no length is specified.
- `VARCHAR`: Variable-length character string declared using VARCHAR(n)
where n is the maximum number of code points. n must have a value
between `1` and `2,147,483,647` (inclusive). Defaults to n=1 if no
length is specified.

**Apache Doris:**
- `CHAR `: Maximum length is `255` characters
- `VARCHAR` : Maximum length is `65,533` characters

**Solution:**
This PR addresses the length constraint mismatch by automatically
converting oversized Paimon VARCHAR/CHAR types to Doris STRING type when
they exceed Doris limits:
- Paimon `VARCHAR` with length > 65,533 → Doris `STRING`
- Paimon `CHAR` with length > 255 → Doris `STRING`

This ensures compatibility while preserving data integrity during type
mapping from Paimon to Doris.
@github-actions github-actions bot requested a review from morrySnow as a code owner August 28, 2025 21:18
@Thearas
Copy link
Contributor

Thearas commented Aug 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 28, 2025
@Thearas
Copy link
Contributor

Thearas commented Aug 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

TPC-H: Total hot run time: 32630 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20a79233ad4c13394e76696ac3cfd5e84a8c73f7, data reload: false

------ Round 1 ----------------------------------
q1	17617	5524	5393	5393
q2	2018	392	293	293
q3	11974	1215	761	761
q4	10219	869	450	450
q5	8262	2353	2123	2123
q6	180	161	132	132
q7	886	757	605	605
q8	9319	1447	1154	1154
q9	5277	4956	4960	4956
q10	6786	2269	1810	1810
q11	466	287	268	268
q12	357	363	207	207
q13	17790	3565	2995	2995
q14	228	227	216	216
q15	525	461	466	461
q16	433	445	387	387
q17	597	845	364	364
q18	6760	6300	6470	6300
q19	1211	943	541	541
q20	346	342	215	215
q21	2915	2196	2015	2015
q22	1059	1028	984	984
Total cold run time: 105225 ms
Total hot run time: 32630 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5513	5450	5413	5413
q2	234	326	237	237
q3	2281	2603	2344	2344
q4	1349	1795	1357	1357
q5	4525	4996	5076	4996
q6	182	163	131	131
q7	2109	1980	1898	1898
q8	2713	2860	2772	2772
q9	7277	7165	7201	7165
q10	3007	3343	2783	2783
q11	565	485	479	479
q12	642	775	645	645
q13	3419	3791	3148	3148
q14	263	290	273	273
q15	524	482	448	448
q16	438	484	453	453
q17	1238	1738	1257	1257
q18	7653	7502	7342	7342
q19	829	1069	1078	1069
q20	2020	2059	1882	1882
q21	5384	5062	4683	4683
q22	1073	1050	995	995
Total cold run time: 53238 ms
Total hot run time: 51770 ms

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

TPC-DS: Total hot run time: 192110 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20a79233ad4c13394e76696ac3cfd5e84a8c73f7, data reload: false

query1	956	393	407	393
query2	6170	1916	1848	1848
query3	8689	200	195	195
query4	33754	23845	23365	23365
query5	3710	590	432	432
query6	283	195	188	188
query7	4212	482	312	312
query8	294	236	234	234
query9	9417	2584	2581	2581
query10	476	309	261	261
query11	18165	15393	15132	15132
query12	166	107	106	106
query13	1554	532	417	417
query14	9011	7278	7575	7278
query15	248	192	180	180
query16	8128	660	532	532
query17	1600	774	591	591
query18	2246	411	321	321
query19	221	194	169	169
query20	132	125	118	118
query21	208	131	106	106
query22	4657	4611	4446	4446
query23	35230	34406	34847	34406
query24	7361	2666	2689	2666
query25	495	491	417	417
query26	1079	246	172	172
query27	2166	486	363	363
query28	5695	2203	2180	2180
query29	599	578	488	488
query30	266	193	163	163
query31	1038	931	810	810
query32	90	62	58	58
query33	528	391	302	302
query34	755	859	518	518
query35	791	809	752	752
query36	1000	1054	971	971
query37	111	92	69	69
query38	4043	3993	3985	3985
query39	1526	1469	1477	1469
query40	215	124	102	102
query41	51	48	47	47
query42	125	108	105	105
query43	510	505	475	475
query44	1311	826	817	817
query45	189	177	186	177
query46	875	1067	651	651
query47	1990	1992	1962	1962
query48	413	425	358	358
query49	774	481	398	398
query50	672	678	428	428
query51	7244	7338	7158	7158
query52	100	99	97	97
query53	232	253	188	188
query54	543	546	478	478
query55	80	79	78	78
query56	267	259	253	253
query57	1272	1259	1228	1228
query58	234	220	218	218
query59	3029	3150	2940	2940
query60	280	281	256	256
query61	108	112	110	110
query62	790	742	686	686
query63	226	192	190	190
query64	4739	1001	634	634
query65	3387	3309	3352	3309
query66	961	426	309	309
query67	16831	15925	15721	15721
query68	7721	805	523	523
query69	479	301	268	268
query70	1142	1071	1107	1071
query71	426	299	264	264
query72	5129	3750	3848	3750
query73	633	745	341	341
query74	10508	9306	8843	8843
query75	3758	3151	2658	2658
query76	3557	1183	759	759
query77	789	355	276	276
query78	10337	10449	9608	9608
query79	5380	860	593	593
query80	625	517	414	414
query81	490	262	220	220
query82	231	119	87	87
query83	162	156	142	142
query84	289	99	84	84
query85	736	373	287	287
query86	356	322	292	292
query87	4380	4298	4178	4178
query88	4271	2397	2375	2375
query89	427	329	291	291
query90	2068	186	188	186
query91	150	157	106	106
query92	63	56	50	50
query93	3715	874	535	535
query94	656	410	314	314
query95	338	278	275	275
query96	482	614	279	279
query97	3165	3267	3170	3170
query98	215	215	197	197
query99	1459	1409	1307	1307
Total cold run time: 298655 ms
Total hot run time: 192110 ms

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

ClickBench: Total hot run time: 28.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20a79233ad4c13394e76696ac3cfd5e84a8c73f7, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.54	0.51	0.50
query6	1.14	0.73	0.74
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.56	0.56	0.55
query11	0.14	0.09	0.10
query12	0.13	0.11	0.11
query13	0.62	0.60	0.60
query14	0.77	0.80	0.78
query15	0.84	0.84	0.83
query16	0.38	0.40	0.36
query17	1.04	1.05	1.05
query18	0.24	0.24	0.22
query19	1.92	1.76	1.71
query20	0.01	0.01	0.00
query21	15.42	0.94	0.59
query22	0.72	0.80	0.61
query23	15.14	1.41	0.60
query24	3.84	0.51	2.38
query25	0.23	0.22	0.05
query26	0.24	0.16	0.13
query27	0.04	0.04	0.05
query28	13.48	0.93	0.43
query29	12.59	3.93	3.25
query30	0.25	0.08	0.06
query31	2.83	0.61	0.37
query32	3.22	0.53	0.45
query33	3.04	2.97	3.09
query34	16.50	5.21	4.52
query35	4.59	4.51	4.52
query36	0.63	0.49	0.47
query37	0.09	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.15 s
Total hot run time: 28.16 s

@vinlee19
Copy link
Contributor

vinlee19 commented Aug 29, 2025

run p0

@vinlee19
Copy link
Contributor

vinlee19 commented Aug 29, 2025

run cloud_p0

@vinlee19
Copy link
Contributor

vinlee19 commented Aug 29, 2025

run external

@morrySnow morrySnow closed this Sep 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @vinlee19 @dataroaring @morrySnow