Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](profile) add catalog info in profile #38283

Merged
merged 2 commits into from
Jul 26, 2024
Merged

[Improvement](profile) add catalog info in profile #38283

merged 2 commits into from
Jul 26, 2024

Conversation

Yulei-Yang
Copy link
Contributor

add catalog in load/query profile
image

    "column_names": [
        "Profile ID",
        "Task Type",
        "Start Time",
        "End Time",
        "Total",
        "Task State",
        "User",
        "Default Catalog",
        "Default Db",
        "Sql Statement"
    ],
    "rows": [
        {
            "Task Type": "QUERY",
            "User": "root",
            "Default Catalog": "internal",
            "Total": "14ms",
            "__hrefPaths": [
                "/query_profile/727f38676ca442bb-b33a77ea8dca569d"
            ],
            "Default Db": "regression_test_query_profile",
            "Profile ID": "727f38676ca442bb-b33a77ea8dca569d",
            "Task State": "EOF",
            "Sql Statement": "select count(*) from test_profile",
            "Start Time": "2024-07-24 03:19:39",
            "End Time": "2024-07-24 03:19:39"
        },

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40100 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d21e6cc072cfafc14f33becfd818335299da6ad0, data reload: false

------ Round 1 ----------------------------------
q1	17790	4490	4377	4377
q2	2360	192	191	191
q3	10513	1227	1109	1109
q4	10194	818	772	772
q5	7588	2690	2684	2684
q6	227	138	137	137
q7	961	602	603	602
q8	9236	2076	2080	2076
q9	8698	6536	6587	6536
q10	8615	3826	3781	3781
q11	459	242	241	241
q12	397	234	226	226
q13	17758	3011	2950	2950
q14	269	248	243	243
q15	540	492	491	491
q16	501	388	386	386
q17	968	668	693	668
q18	8207	7414	7449	7414
q19	7472	1446	1414	1414
q20	679	331	336	331
q21	4974	3189	3293	3189
q22	350	282	290	282
Total cold run time: 118756 ms
Total hot run time: 40100 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4435	4290	4235	4235
q2	385	275	275	275
q3	2986	2778	2747	2747
q4	1884	1646	1663	1646
q5	5301	5303	5363	5303
q6	218	131	134	131
q7	2129	1766	1688	1688
q8	3220	3367	3327	3327
q9	8425	8467	8405	8405
q10	3929	3663	3661	3661
q11	599	489	493	489
q12	765	624	605	605
q13	17496	2970	2997	2970
q14	322	276	273	273
q15	525	481	467	467
q16	482	437	432	432
q17	1805	1487	1473	1473
q18	7669	7612	7332	7332
q19	1655	1515	1481	1481
q20	2011	1821	1798	1798
q21	4871	4704	4696	4696
q22	570	465	508	465
Total cold run time: 71682 ms
Total hot run time: 53899 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172977 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d21e6cc072cfafc14f33becfd818335299da6ad0, data reload: false

query1	913	385	365	365
query2	6452	1908	1915	1908
query3	6649	217	224	217
query4	28188	17292	17220	17220
query5	4174	466	491	466
query6	287	165	182	165
query7	4589	283	275	275
query8	238	200	192	192
query9	8368	2418	2384	2384
query10	427	295	264	264
query11	12928	9963	10054	9963
query12	140	89	83	83
query13	1633	370	360	360
query14	10358	7680	7936	7680
query15	246	165	166	165
query16	7833	419	454	419
query17	1248	557	523	523
query18	1961	277	278	277
query19	195	160	155	155
query20	90	88	83	83
query21	204	127	126	126
query22	4389	4001	3911	3911
query23	33813	33174	33770	33174
query24	12305	2934	2892	2892
query25	773	370	373	370
query26	1806	148	146	146
query27	2939	269	274	269
query28	7417	2004	1994	1994
query29	1192	629	620	620
query30	294	151	152	151
query31	975	746	735	735
query32	93	55	52	52
query33	755	342	333	333
query34	924	475	492	475
query35	852	730	742	730
query36	1170	913	956	913
query37	275	88	83	83
query38	2895	2809	2805	2805
query39	875	826	801	801
query40	280	126	125	125
query41	49	47	48	47
query42	124	101	107	101
query43	527	487	484	484
query44	1236	726	737	726
query45	194	164	166	164
query46	1113	738	744	738
query47	1830	1752	1778	1752
query48	371	297	286	286
query49	1161	466	408	408
query50	802	395	391	391
query51	6804	6752	6682	6682
query52	106	90	95	90
query53	362	298	289	289
query54	936	454	445	445
query55	74	76	75	75
query56	293	279	278	278
query57	1214	1042	1051	1042
query58	273	246	261	246
query59	2948	2695	3017	2695
query60	310	277	280	277
query61	134	94	95	94
query62	806	623	646	623
query63	329	294	294	294
query64	10523	2269	1695	1695
query65	3194	3084	3177	3084
query66	1203	335	333	333
query67	15354	14910	15017	14910
query68	4558	541	558	541
query69	475	355	365	355
query70	1201	1128	1107	1107
query71	384	289	275	275
query72	7122	5869	5450	5450
query73	755	332	327	327
query74	6117	5648	5646	5646
query75	3411	2693	2716	2693
query76	2825	925	911	911
query77	445	309	313	309
query78	11007	8993	8903	8903
query79	2856	527	531	527
query80	2038	475	473	473
query81	584	230	223	223
query82	710	136	135	135
query83	277	168	168	168
query84	281	84	86	84
query85	2041	326	312	312
query86	484	329	310	310
query87	3287	3079	3109	3079
query88	4495	2459	2399	2399
query89	485	380	382	380
query90	1764	197	189	189
query91	135	106	102	102
query92	63	49	49	49
query93	3992	516	509	509
query94	850	312	299	299
query95	404	317	322	317
query96	616	272	273	272
query97	3208	3074	3061	3061
query98	222	201	190	190
query99	1566	1292	1264	1264
Total cold run time: 292071 ms
Total hot run time: 172977 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d21e6cc072cfafc14f33becfd818335299da6ad0, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.06	0.05
query4	1.65	0.07	0.07
query5	0.49	0.48	0.50
query6	1.13	0.72	0.72
query7	0.01	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.47
query10	0.54	0.55	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.59
query14	0.76	0.77	0.77
query15	0.86	0.82	0.82
query16	0.37	0.37	0.36
query17	1.00	0.97	0.97
query18	0.23	0.23	0.23
query19	1.82	1.68	1.71
query20	0.01	0.01	0.01
query21	15.40	0.78	0.65
query22	3.36	7.02	2.43
query23	18.75	1.32	1.30
query24	2.16	0.23	0.22
query25	0.15	0.10	0.08
query26	0.30	0.21	0.21
query27	0.45	0.23	0.24
query28	13.21	1.02	1.00
query29	12.59	3.34	3.33
query30	0.25	0.07	0.06
query31	2.89	0.40	0.40
query32	3.25	0.47	0.47
query33	2.92	2.91	2.87
query34	17.26	4.40	4.35
query35	4.45	4.44	4.46
query36	0.66	0.47	0.49
query37	0.20	0.16	0.15
query38	0.15	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.53 s
Total hot run time: 31.22 s

@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b79efa514ff06bf84b6975d399efc897d8a8380d, data reload: false

------ Round 1 ----------------------------------
q1	17622	4395	4265	4265
q2	2019	198	182	182
q3	10531	1259	1066	1066
q4	10807	799	760	760
q5	7568	2736	2721	2721
q6	225	138	139	138
q7	976	610	611	610
q8	9590	2101	2117	2101
q9	9002	6633	6582	6582
q10	8671	3739	3749	3739
q11	452	244	229	229
q12	395	226	220	220
q13	17785	2991	2973	2973
q14	272	233	240	233
q15	525	474	495	474
q16	503	381	377	377
q17	976	691	681	681
q18	8094	7609	7421	7421
q19	2048	1346	1445	1346
q20	641	321	316	316
q21	5003	3238	3265	3238
q22	337	279	280	279
Total cold run time: 114042 ms
Total hot run time: 39951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4410	4334	4243	4243
q2	386	277	276	276
q3	2972	2767	2788	2767
q4	1877	1599	1657	1599
q5	5299	5328	5310	5310
q6	228	128	128	128
q7	2115	1696	1738	1696
q8	3171	3383	3311	3311
q9	8410	8342	8424	8342
q10	3908	3692	3717	3692
q11	592	485	485	485
q12	784	618	606	606
q13	16341	2992	3015	2992
q14	315	276	275	275
q15	518	472	475	472
q16	489	413	423	413
q17	1771	1498	1455	1455
q18	7649	7653	7360	7360
q19	2006	1425	1442	1425
q20	2007	1779	1797	1779
q21	4971	4755	4637	4637
q22	574	504	515	504
Total cold run time: 70793 ms
Total hot run time: 53767 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172252 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b79efa514ff06bf84b6975d399efc897d8a8380d, data reload: false

query1	909	371	366	366
query2	6449	1838	1753	1753
query3	6649	207	233	207
query4	27836	17554	17422	17422
query5	4195	477	489	477
query6	272	163	165	163
query7	4593	292	292	292
query8	252	204	194	194
query9	8572	2414	2384	2384
query10	443	281	270	270
query11	11686	10024	10117	10024
query12	139	85	79	79
query13	1660	386	366	366
query14	10291	7668	6218	6218
query15	215	168	168	168
query16	7825	422	519	422
query17	1534	562	544	544
query18	1924	283	284	283
query19	205	158	158	158
query20	90	89	86	86
query21	204	132	128	128
query22	4446	4075	4135	4075
query23	33842	35718	33406	33406
query24	12143	2859	2819	2819
query25	525	379	389	379
query26	1110	158	160	158
query27	2929	287	279	279
query28	7573	2022	1999	1999
query29	885	644	621	621
query30	297	155	154	154
query31	977	735	768	735
query32	105	56	56	56
query33	776	340	345	340
query34	901	474	485	474
query35	895	748	722	722
query36	1097	962	935	935
query37	217	80	80	80
query38	2903	2798	2866	2798
query39	855	827	838	827
query40	285	125	122	122
query41	52	48	48	48
query42	116	99	100	99
query43	498	475	462	462
query44	1270	737	737	737
query45	196	163	166	163
query46	1088	760	735	735
query47	1858	1786	1799	1786
query48	373	299	298	298
query49	1201	423	420	420
query50	783	395	391	391
query51	6680	6702	6592	6592
query52	106	93	99	93
query53	367	299	302	299
query54	871	463	472	463
query55	74	77	76	76
query56	319	280	278	278
query57	1132	1053	1043	1043
query58	372	246	249	246
query59	2860	2619	2666	2619
query60	309	275	272	272
query61	101	96	91	91
query62	830	652	657	652
query63	323	293	295	293
query64	10343	2247	1679	1679
query65	3183	3130	3109	3109
query66	1225	334	323	323
query67	15709	14932	14958	14932
query68	9659	564	559	559
query69	753	435	382	382
query70	1386	1133	1166	1133
query71	579	282	278	278
query72	8418	5442	5722	5442
query73	2242	323	324	323
query74	6257	5717	5751	5717
query75	6245	2680	2696	2680
query76	5637	1008	934	934
query77	792	312	324	312
query78	11803	9543	9082	9082
query79	10736	528	532	528
query80	1213	481	463	463
query81	578	222	227	222
query82	245	135	136	135
query83	348	164	162	162
query84	271	86	85	85
query85	748	304	295	295
query86	377	314	317	314
query87	3360	3077	3128	3077
query88	4963	2398	2376	2376
query89	492	389	385	385
query90	2276	193	192	192
query91	127	104	100	100
query92	67	55	49	49
query93	4048	514	511	511
query94	1512	282	297	282
query95	398	308	317	308
query96	620	269	270	269
query97	3180	3015	3018	3015
query98	211	207	190	190
query99	1489	1287	1270	1270
Total cold run time: 311973 ms
Total hot run time: 172252 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b79efa514ff06bf84b6975d399efc897d8a8380d, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.51	0.49	0.49
query6	1.24	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.54	0.55	0.55
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.75	0.79	0.77
query15	0.86	0.82	0.81
query16	0.36	0.37	0.36
query17	0.96	0.95	1.02
query18	0.23	0.22	0.21
query19	1.88	1.82	1.80
query20	0.01	0.00	0.01
query21	15.41	0.78	0.65
query22	4.01	6.86	2.52
query23	18.23	1.43	1.40
query24	2.12	0.25	0.23
query25	0.15	0.09	0.08
query26	0.30	0.20	0.20
query27	0.46	0.23	0.24
query28	13.17	1.01	0.99
query29	12.60	3.35	3.34
query30	0.26	0.06	0.06
query31	2.87	0.38	0.38
query32	3.31	0.48	0.49
query33	2.91	2.95	2.96
query34	16.89	4.38	4.40
query35	4.39	4.52	4.43
query36	0.65	0.47	0.47
query37	0.18	0.15	0.16
query38	0.16	0.16	0.14
query39	0.05	0.03	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 109.38 s
Total hot run time: 31.59 s

@Yulei-Yang
Copy link
Contributor Author

run feut

@Yulei-Yang
Copy link
Contributor Author

run cloud_p0

@Yulei-Yang Yulei-Yang mentioned this pull request Jul 24, 2024
Merged
@Yulei-Yang
Copy link
Contributor Author

run cloud_p0

@Yulei-Yang Yulei-Yang mentioned this pull request Jul 24, 2024
Merged
@Yulei-Yang
Copy link
Contributor Author

run cloud_p0

1 similar comment
@Yulei-Yang
Copy link
Contributor Author

run cloud_p0

@Yulei-Yang Yulei-Yang mentioned this pull request Jul 24, 2024
Merged
@Yulei-Yang
Copy link
Contributor Author

run cloud_p0

yiguolei pushed a commit that referenced this pull request Jul 25, 2024
@Yulei-Yang
[Improvement](profile) add catalog info in profile (#38302)
c23228e 
bp #38283
liutang123
liutang123 approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@liutang123 liutang123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

nextdreamblue
nextdreamblue approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@nextdreamblue nextdreamblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liutang123 liutang123 merged commit 53ceaeb into apache:master Jul 26, 2024
28 of 30 checks passed
liutang123 pushed a commit that referenced this pull request Jul 26, 2024
@Yulei-Yang
[Improvement](profile) add catalog info in profile (#38298)
bdbbcb7 
bp #38283
liutang123 pushed a commit that referenced this pull request Jul 26, 2024
@Yulei-Yang
[Improvement](profile) add catalog info in profile (#38326)
61e5874 
bp #38283 add `Default Catalog` column in load/query profile summary
@Yulei-Yang Yulei-Yang deleted the add_catalog_for_profile branch July 26, 2024 02:28
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@Yulei-Yang @dataroaring
[Improvement](profile) add catalog info in profile (#38283)
a837b4f 
add `Default Catalog` column in load/query profile summary
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@Yulei-Yang @dataroaring
[Improvement](profile) add catalog info in profile (#38283)
157499b 
add `Default Catalog` column in load/query profile summary
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@Yulei-Yang
[Improvement](profile) add catalog info in profile (apache#38298)
63dfe3a 
bp apache#38283
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liutang123 liutang123 liutang123 approved these changes

@nextdreamblue nextdreamblue nextdreamblue approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Yulei-Yang @doris-robot @liutang123 @nextdreamblue @yiguolei @dataroaring