Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](load) Fix using uint32 for tablet id overflow #31532

Merged
merged 1 commit into from
Feb 28, 2024
Merged

[fix](load) Fix using uint32 for tablet id overflow #31532

merged 1 commit into from
Feb 28, 2024

Conversation

liaoxin01
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

[INTERNAL_ERROR][INTERNAL_ERROR]close wait failed coz rpc error. VNodeChannel[4310428608-10003], load_id=c478455147aeb7f-847eb9b9965435a7, txn_id=42764708, node=10.86.1.83:8060, add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (10.86.1.83)[INTERNAL_ERROR]unknown tablet to append data, tablet=15461331

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@liaoxin01
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38019 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c68475eebda09b3c32794775d1cc9036db9e76eb, data reload: false

------ Round 1 ----------------------------------
q1	18133	4183	4120	4120
q2	2603	146	140	140
q3	11000	983	966	966
q4	5041	971	998	971
q5	7973	3000	3075	3000
q6	185	126	122	122
q7	1294	788	777	777
q8	9638	1968	2072	1968
q9	7223	6267	6263	6263
q10	8178	2481	2489	2481
q11	421	202	216	202
q12	798	318	309	309
q13	17973	3263	3207	3207
q14	285	262	251	251
q15	530	499	487	487
q16	470	418	412	412
q17	941	892	886	886
q18	6577	6126	5811	5811
q19	1569	1510	1511	1510
q20	549	287	279	279
q21	6781	3583	3577	3577
q22	801	283	280	280
Total cold run time: 108963 ms
Total hot run time: 38019 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4008	3983	4007	3983
q2	300	215	211	211
q3	3001	2927	2949	2927
q4	1868	1813	1805	1805
q5	5084	5126	5125	5125
q6	200	114	114	114
q7	2130	1699	1717	1699
q8	3105	3182	3197	3182
q9	8258	8237	8267	8237
q10	6213	3624	3628	3624
q11	517	433	437	433
q12	713	517	521	517
q13	3826	3069	3061	3061
q14	276	242	256	242
q15	532	492	487	487
q16	502	466	476	466
q17	1700	1678	1677	1677
q18	7839	7535	7391	7391
q19	1630	1643	1621	1621
q20	2094	1910	1907	1907
q21	4757	4640	4593	4593
q22	513	455	462	455
Total cold run time: 59066 ms
Total hot run time: 53757 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169506 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c68475eebda09b3c32794775d1cc9036db9e76eb, data reload: false

query1	941	337	335	335
query2	6521	1746	1658	1658
query3	6720	210	210	210
query4	23408	20590	20626	20590
query5	4223	371	373	371
query6	258	170	160	160
query7	4618	301	303	301
query8	274	194	196	194
query9	8412	2260	2243	2243
query10	417	211	218	211
query11	14675	14164	14180	14164
query12	132	83	82	82
query13	1653	426	437	426
query14	8277	6637	6532	6532
query15	197	178	186	178
query16	7089	262	248	248
query17	1043	532	502	502
query18	1894	271	265	265
query19	193	146	144	144
query20	87	81	86	81
query21	198	132	117	117
query22	4789	4668	4547	4547
query23	30571	29821	29951	29821
query24	11824	3131	3119	3119
query25	654	356	345	345
query26	1782	152	158	152
query27	3018	317	324	317
query28	7008	1812	1804	1804
query29	1159	563	566	563
query30	284	132	139	132
query31	869	685	706	685
query32	94	58	56	56
query33	725	220	219	219
query34	1046	482	494	482
query35	836	732	748	732
query36	926	881	849	849
query37	134	61	61	61
query38	3152	3007	2960	2960
query39	1318	1290	1261	1261
query40	280	101	104	101
query41	38	36	37	36
query42	103	98	101	98
query43	453	425	427	425
query44	1055	708	728	708
query45	197	183	172	172
query46	1036	786	792	786
query47	1543	1453	1495	1453
query48	434	346	343	343
query49	1123	285	296	285
query50	783	383	388	383
query51	4377	4341	4260	4260
query52	97	93	89	89
query53	330	266	264	264
query54	299	226	224	224
query55	87	80	86	80
query56	224	197	194	194
query57	959	883	872	872
query58	217	201	186	186
query59	2260	2085	2101	2085
query60	242	215	214	214
query61	90	84	82	82
query62	607	367	360	360
query63	280	263	251	251
query64	6578	3047	3109	3047
query65	3221	3209	3224	3209
query66	1443	327	310	310
query67	14490	14371	14130	14130
query68	5004	558	572	558
query69	535	364	378	364
query70	1276	1225	1206	1206
query71	411	255	268	255
query72	6230	2785	2656	2656
query73	707	320	334	320
query74	6304	5960	5959	5959
query75	3075	2455	2410	2410
query76	2789	1025	1141	1025
query77	352	227	223	223
query78	9068	8496	8492	8492
query79	1125	531	523	523
query80	813	357	341	341
query81	465	194	193	193
query82	1305	84	84	84
query83	205	120	121	120
query84	278	80	82	80
query85	1189	340	330	330
query86	337	297	303	297
query87	3349	3109	3091	3091
query88	2793	2332	2323	2323
query89	388	335	324	324
query90	2037	164	169	164
query91	151	119	122	119
query92	56	47	51	47
query93	1031	521	510	510
query94	1113	180	177	177
query95	437	332	348	332
query96	573	270	265	265
query97	4248	4160	4151	4151
query98	218	202	186	186
query99	1043	617	691	617
Total cold run time: 263170 ms
Total hot run time: 169506 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.69% (8548/23952)
Line Coverage: 27.49% (69386/252411)
Region Coverage: 26.65% (35974/135007)
Branch Coverage: 23.46% (18389/78394)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c68475eebda09b3c32794775d1cc9036db9e76eb_c68475eebda09b3c32794775d1cc9036db9e76eb/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c68475eebda09b3c32794775d1cc9036db9e76eb, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.52	0.52	0.52
query6	1.25	0.66	0.69
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.53	0.53
query10	0.56	0.56	0.58
query11	0.13	0.10	0.09
query12	0.12	0.10	0.10
query13	0.63	0.61	0.61
query14	0.79	0.79	0.82
query15	0.84	0.82	0.80
query16	0.37	0.37	0.39
query17	0.99	1.00	0.96
query18	0.27	0.25	0.25
query19	1.77	1.74	1.73
query20	0.01	0.01	0.02
query21	15.41	0.62	0.62
query22	2.58	3.35	2.45
query23	17.64	1.06	1.04
query24	2.16	0.24	0.12
query25	0.23	0.12	0.05
query26	0.16	0.15	0.15
query27	0.05	0.04	0.04
query28	12.81	0.83	0.83
query29	12.60	3.24	3.31
query30	0.57	0.54	0.54
query31	2.79	0.35	0.35
query32	3.36	0.48	0.48
query33	3.16	3.22	3.21
query34	15.36	4.73	4.73
query35	4.74	4.71	4.71
query36	1.13	1.07	1.07
query37	0.06	0.04	0.04
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.18	0.13	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 106.01 s
Total hot run time: 32.1 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c68475eebda09b3c32794775d1cc9036db9e76eb with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.8 seconds inserted 10000000 Rows, about 595K ops/s

dataroaring
dataroaring approved these changes Feb 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Feb 28, 2024
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit aa366f6 into apache:master Feb 28, 2024
28 of 31 checks passed
yiguolei pushed a commit that referenced this pull request Feb 29, 2024
@liaoxin01 @Doris-Extras
[fix](load) Fix using uint32 for tablet id overflow (#31532)
8c570c2
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Mar 4, 2024
@liaoxin01
[fix](load) Fix using uint32 for tablet id overflow (apache#31532)
31c2eab
@liaoxin01 liaoxin01 mentioned this pull request Mar 4, 2024
Merged
yiguolei pushed a commit that referenced this pull request Mar 4, 2024
@liaoxin01
[fix](load) Fix using uint32 for tablet id overflow (#31532) (#31758)
02b5923
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@liaoxin01
[fix](load) Fix using uint32 for tablet id overflow (apache#31532) (a…
195084a 
…pache#31758)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sollhui sollhui sollhui approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.6-merged dev/2.1.0 p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@liaoxin01 @doris-robot @sollhui @dataroaring @xiaokang @yiguolei