Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Nereids) result type of add precision is 1 more than expected #27136

Merged
merged 1 commit into from
Nov 17, 2023

Conversation

morrySnow
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 10a13548d81c1a079affdc062eb90adb0d65a21a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4924	4644	4633	4633
q2	355	167	159	159
q3	2034	1948	1951	1948
q4	1385	1240	1232	1232
q5	3948	3935	4059	3935
q6	251	129	133	129
q7	1387	891	879	879
q8	2759	2773	2773	2773
q9	9774	9855	9668	9668
q10	3461	3546	3512	3512
q11	376	254	243	243
q12	439	296	299	296
q13	4538	3834	3831	3831
q14	319	284	288	284
q15	581	541	521	521
q16	676	580	583	580
q17	1132	985	927	927
q18	7782	7497	7384	7384
q19	1670	1709	1668	1668
q20	515	293	306	293
q21	4601	3986	4043	3986
q22	477	369	376	369
Total cold run time: 53384 ms
Total hot run time: 49250 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4614	4593	4578	4578
q2	340	221	275	221
q3	4005	4015	3991	3991
q4	2700	2700	2693	2693
q5	9566	9590	9540	9540
q6	246	123	125	123
q7	2615	2303	2295	2295
q8	4481	4449	4492	4449
q9	13198	13066	13131	13066
q10	4083	4178	4165	4165
q11	807	692	677	677
q12	977	830	809	809
q13	4303	3588	3561	3561
q14	385	359	339	339
q15	581	524	526	524
q16	722	661	667	661
q17	3836	3929	3875	3875
q18	9724	9015	9115	9015
q19	1825	1792	1776	1776
q20	2371	2062	2048	2048
q21	8855	8438	8638	8438
q22	900	815	792	792
Total cold run time: 81134 ms
Total hot run time: 77636 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.24 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17098789385 Bytes

@yiguolei
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 6f570326bfa6e08d9cfbea2a8defd213549343c4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4960	4646	4687	4646
q2	356	144	157	144
q3	2034	1981	1905	1905
q4	1385	1233	1217	1217
q5	3943	3942	4021	3942
q6	249	130	132	130
q7	1378	876	877	876
q8	2767	2799	2775	2775
q9	9865	9609	9552	9552
q10	3481	3523	3525	3523
q11	378	247	237	237
q12	437	297	301	297
q13	4545	3786	3800	3786
q14	317	288	300	288
q15	579	528	524	524
q16	670	593	583	583
q17	1144	965	927	927
q18	7951	7419	7457	7419
q19	1678	1681	1679	1679
q20	533	325	323	323
q21	4459	3992	3999	3992
q22	475	377	373	373
Total cold run time: 53584 ms
Total hot run time: 49138 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4593	4596	4591	4591
q2	335	223	247	223
q3	4015	4002	3985	3985
q4	2701	2690	2703	2690
q5	9710	9669	9760	9669
q6	242	127	124	124
q7	2590	2314	2255	2255
q8	4433	4410	4449	4410
q9	13230	13195	13121	13121
q10	4078	4174	4187	4174
q11	779	699	700	699
q12	986	816	812	812
q13	4279	3609	3613	3609
q14	382	358	350	350
q15	582	517	522	517
q16	734	662	683	662
q17	3952	3950	3846	3846
q18	9589	9077	8912	8912
q19	1829	1790	1782	1782
q20	2414	2070	2037	2037
q21	8776	8774	8638	8638
q22	877	795	768	768
Total cold run time: 81106 ms
Total hot run time: 77874 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.73 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 17 seconds loaded 2358488459 Bytes, about 132 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17099413218 Bytes

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit fa7e1b7 into apache:master Nov 17, 2023
26 of 28 checks passed
@xiaokang xiaokang added the p0_b label Nov 22, 2023
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Nov 22, 2023
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Nov 27, 2023
eldenmoon added a commit that referenced this pull request Nov 27, 2023
* [fix](stats) Fix update rows for unique table didn't get updated properly #26968 (#27337)

* [FIX](jsonb) fix jsonb in predict column #27325 (#27424)

* [fix](fe) slots in having clause should be set to need materialized(#27412) (#27429)

* [Bug](insert)fix insert wrong data on mv when stmt have multiple values (#27297) (#27382)

fix insert wrong data on mv when stmt have multiple values

* [fix](fe ut) Fix OlapQueryCacheTest failed (#27305) (#27406)

1.
```
java.lang.NullPointerException: null
        at org.apache.doris.catalog.Env.getCurrentSystemInfo(Env.java:793) ~[classes/:?]
        at org.apache.doris.qe.SimpleScheduler$UpdateBlacklistThread.run(SimpleScheduler.java:206) ~[classes/:?]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

java.lang.NullPointerException
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:226)
```

2.
```
[ERROR] testSqlCacheKeyWithNestedViewForNereids  Time elapsed: 1.962 s  <<< FAILURE!
java.lang.AssertionError: SELECT command denied to user 'testCluster:testUser'@'192.168.1.1' for table 'internal: testCluster:testDb: appevent'
	at org.apache.doris.qe.OlapQueryCacheTest.parseSqlByNereids(OlapQueryCacheTest.java:579)
	at org.apache.doris.qe.OlapQueryCacheTest.testSqlCacheKeyWithNestedViewForNereids(OlapQueryCacheTest.java:1338)
```

3.
```
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.63 s <<< FAILURE! - in org.apache.doris.qe.OlapQueryCacheTest
[ERROR] testCacheModeTable  Time elapsed: 1.657 s  <<< ERROR!
java.lang.IllegalArgumentException: Value of type org.apache.doris.qe.QueryState incompatible with return type org.apache.doris.system.SystemInfoService of org.apache.doris.catalog.Env#getCurrentSystemInfo()
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:156)
```

* [regression test](schema change) add some schema change regression cases (#27112) (#27418)

* [fix](Nereids) result type of add precision is 1 more than expected (#27136) (#27426)

* [fix](Nereids): fill miss slot in having subquery (#27177) (#27394)

* [fix](memory) Fix make_top_consumption_snapshots heap-use-after-free #27434 (#27465)

* [fix](function) make TIMESTAMP function DEPEND_ON_ARGUMENT (#27343) (#27458)

* [fix](test) order by clause in test_map(#27390) (#27391)

pick #27390

* [performance](Planner): optimize getStringValue() in DateLiteral (#27363) (#27470)

- reduce cost of `getStringValue()`
- original code don't consider `microsecond` part in `getStringValue()`

(cherry picked from commit 044a295)

* [Chore](pick) do not push down agg on aggregate column (#27356) (#27498)

* [fix](stats) table not exists error msg not print objects name #27074 (#27463)

* [improve](nereids) support agg function of count(const value) pushdown #26677 (#27499)

support sql: select count(1)-count(not null) from table, the agg of count could push down.

* [test](fe-ut) fix unstable MysqlServerTest (#27459)

Need to find a unbind port for MysqlServerTest

* [opt](MergedIO) no need to merge large columns (#27315) (#27497)

1. Fix a profile bug of `MergeRangeFileReader`, and add a profile `ApplyBytes` to show the total bytes  of ranges.
2. There's no need to merge large columns, because `MergeRangeFileReader` will increase the copy time.

* [improvement](drop tablet)  impr gc shutdown tablet lock (#26151) (#27478)

* [doc](stats) SQL manual for stats (#27461)

* [chore](merge-on-write) disable rowid conversion check for mow table by default (#27482) (#27508)

* [fix](regression)Fix hive p2 case (#27466) (#27511)

* [fix](statistics)Fix auto analyze remove finished job bug #27486 (#27510)

* [Bug](bitmap) Fix heap-use-after-free in the bitmap functions #27411 (#27521)

* [Pick](nereids) Pick: partition prune fails in case of NOT expression (#27047) (#27507)

* [fix](clone) Fix engine_clone file exist (#27361) (#27536)

* [chore](case) adjust timeout of broker load case #27540

* Fix auto analyze doesn't filter unsupported type bug. (#27547)

Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
backport #27559

* [chore](fe plugin) Upgrade dependency to doris 2.0-SNAPSHOT #27522 (#27558)

* [Bug](materialized-view) add limitation for duplicate expr on materialized view (#27523) (#27562)

* [fix](planner)join node should output required slot from parent node #27526 (#27551)

* [branch-2.0](hive) enable hive view by default (#27550)

* [pick](nereids) adjust bc join and shuffle join #27113 (#27566)

* [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27567)

---------

Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: Jack Drogon <jack.xsuperman@gmail.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
@morrySnow morrySnow deleted the fix_decimal_add branch December 27, 2023 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_b reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants