Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. #27563

Merged
merged 1 commit into from
Nov 25, 2023
Merged

[Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. #27563

merged 1 commit into from
Nov 25, 2023

Conversation

kaka11chen
Copy link
Contributor

Proposed changes

Issue:

2023-11-24 17:59:16,171 WARN (mysql-nio-pool-0|236) [HiveScanNode.getSplits():211] get file split failed for table: lineitem
org.apache.doris.datasource.CacheException: failed to get input splits for write ids acid.lineitem:9223372036854775807:9223372036854775807:: in catalog regression1
        at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFilesByTransaction(HiveMetaStoreCache.java:863) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.HiveScanNode.getFileSplitByTransaction(HiveScanNode.java:308) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.HiveScanNode.getFileSplitByPartitions(HiveScanNode.java:223) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.HiveScanNode.getSplits(HiveScanNode.java:206) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:270) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.FileQueryScanNode.doFinalize(FileQueryScanNode.java:229) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.util.Utils.execWithUncheckedException(Utils.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:522) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan.accept(PhysicalFileScan.java:98) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:1568) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalProject.accept(PhysicalProject.java:105) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:784) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.accept(PhysicalHashAggregate.java:185) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:265) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute.accept(PhysicalDistribute.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:784) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.accept(PhysicalHashAggregate.java:185) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:355) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:70) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:239) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:134) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:580) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:441) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:358) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:511) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:762) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.fs.Path.<init>(Path.java:150) ~[hadoop-common-3.3.6.jar:?]
        at org.apache.hadoop.fs.Path.<init>(Path.java:129) ~[hadoop-common-3.3.6.jar:?]
        at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFilesByTransaction(HiveMetaStoreCache.java:805) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 37 more

Solution

Fix NPE when query empty hive transactional table.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the fix_empty_hive_transactional_table branch from b5dd022 to 44eb203 Compare November 24, 2023 12:20
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.83 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17100689076 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 44eb2032557155872f57c793cf431f80b2288dca, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4905	4636	4630	4630
q2	365	141	140	140
q3	2011	1898	1896	1896
q4	1383	1258	1236	1236
q5	3993	3922	3973	3922
q6	255	128	131	128
q7	1397	885	869	869
q8	2784	2789	2757	2757
q9	9644	9544	9403	9403
q10	3473	3524	3544	3524
q11	375	240	251	240
q12	435	293	292	292
q13	4577	3809	3805	3805
q14	321	290	297	290
q15	589	525	511	511
q16	660	581	582	581
q17	1134	939	907	907
q18	7845	7513	7447	7447
q19	1667	1689	1658	1658
q20	537	318	292	292
q21	4409	3933	4017	3933
q22	484	380	379	379
Total cold run time: 53243 ms
Total hot run time: 48840 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4617	4604	4572	4572
q2	351	244	260	244
q3	4040	4027	4025	4025
q4	2718	2700	2708	2700
q5	9860	9904	9914	9904
q6	248	119	124	119
q7	3023	2510	2459	2459
q8	4466	4438	4457	4438
q9	12990	12839	12834	12834
q10	4093	4187	4148	4148
q11	769	661	664	661
q12	971	807	830	807
q13	4313	3615	3574	3574
q14	381	369	342	342
q15	573	518	526	518
q16	738	655	700	655
q17	3847	3892	3827	3827
q18	9690	9144	9004	9004
q19	1816	1779	1800	1779
q20	2394	2046	2025	2025
q21	8812	8519	8521	8519
q22	920	852	778	778
Total cold run time: 81630 ms
Total hot run time: 77932 ms

morningman
morningman approved these changes Nov 24, 2023
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@kaka11chen kaka11chen mentioned this pull request Nov 24, 2023
Merged
@wm1581066 wm1581066 added dev/2.0.4 usercase Important user case type label labels Nov 24, 2023
@yiguolei yiguolei merged commit cc395f5 into apache:master Nov 25, 2023
27 of 28 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@kaka11chen @seawinde
[Fix](hive-transactional-table) Fix NPE when query empty hive transac…
2522bed 
…tional table. (apache#27563)
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@kaka11chen
[Fix](hive-transactional-table) Fix NPE when query empty hive transac…
e288fef 
…tional table. (apache#27563)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jibing-Li Jibing-Li Jibing-Li approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.3-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@kaka11chen @doris-robot @morningman @Jibing-Li @yiguolei @wm1581066