-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDFS-17531. RBF: Aynchronous router RPC. #7308
base: trunk
Are you sure you want to change the base?
Conversation
💔 -1 overall
This message was automatically generated. |
Hi, all, the failed UT is not related to this pr. You can see that another pipeline is successful. https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-7301/1/pipeline |
@KeeProMise we need to fix checkstyle first and trigger Yetus again, it is better to collect one green result. |
616d707
to
15cc381
Compare
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
@KeeProMise @hfutatzhanghb When is the ARR feature expected to be merged into the trunk? Is there a specific plan for it? We are currently working on upgrading to JDK17 and migrating from JUnit4 to JUnit5. The upgrade work is mostly complete locally, and we are in the process of pushing the PR to the trunk. We plan to start upgrading the HDFS-related unit tests next week. If the feature is still under development, we may need to backport our upgrade PR. @KeeProMise If you initiate a vote, I will give it a +1. cc: @Hexiaoqiao |
Hi, slfan1989,
Currently, this feature has been developed and all pipelines have passed.
It is waiting for community voting before being merged into trunk.
|
@slfan1989 @Hexiaoqiao @KeeProMise Sir, this ARR feature has been completed generally and any further optimizations can be pushed forward in HDFS-17716. I think we can initial a vote for megeing ARR into trunk currently. |
I think sub PRs had already been reviewed, no need to review details again. Just leave some high level comments here
|
Hi @ferhui
|
Hi, @slfan1989 Please help check these two failed unit tests https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7378/1/testReport/ . They seem to be related to this PR.HADOOP-19415. [JDK17] Upgrade JUnit from 4 to 5 in hadoop-common Part1. (#7339) |
b47e11e
to
82bccee
Compare
💔 -1 overall
This message was automatically generated. |
Hi, all, the two failed unit tests are not related to this change. For details, please refer to #7378 |
@KeeProMise The issue you reported was caused by my pr. To resolve the problem quickly, I plan to submit a fast fix PR without waiting for the merge of #7375. I have submitted a fix pr(#7383) . |
@KeeProMise #7383 has been submitted to the trunk, and we can trigger the compilation again. cc: @Hexiaoqiao |
For RBF: Asynchronous Router RPC, I give a +1. I will respond in the DISCUSS email thread and participate in the VOTE process. I sincerely hope this feature gets merged successfully. Once again, I appreciate your continuous efforts—wishing you all the best! If the new build is successful, I will approve this PR. cc: @Hexiaoqiao |
…easier. (#6868). Contributed by Jian Zhang. Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…ny. (#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…an Zhang. Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…. Contributed by Archie73. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…Contributed by Wenqi Li. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…ontributed by hfutatzhanghb. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
… Contributed by hfutatzhanghb. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…rts asynchronous rpc. (#7159). Contributed by Jian Zhang. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: Jian Zhang <keepromise@apache.org>
…tributed by hfutatzhanghb. Reviewed-by: Jian Zhang <keepromise@apache.org>
…age. (#7184). Contributed by Jian Zhang.
…tatzhanghb. Reviewed-by: Jian Zhang <keepromise@apache.org>
Co-authored-by: Jian Zhang <keepromise@apache.org>
…nchrony. (#7139). Contributed by hfutatzhanghb. Co-authored-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…d by hfutatzhanghb. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
… Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
82bccee
to
ad83682
Compare
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did some test work referring to the documentation.
- Compiled source codes, built local cluster and the async feature worked fine.
- it is disabled by default
- can increase or decrease the thread number by changing the related configurations
The feature is as described in the documentation. Great work, +1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. +1.
Description of PR
seeAlso: https://issues.apache.org/jira/browse/HDFS-17531
I. Overview
The asynchronous router aims to address the performance bottleneck issues of the synchronous router in high - concurrency and multi - nameservices scenarios. By introducing an asynchronous processing mechanism, it optimizes the request handling process, improves the system's concurrency ability and resource utilization, and is particularly suitable for the federated scenarios where multiple downstream services (NS) need to be processed.
II. Problems of the Synchronous Router
III. Design and Improvements of the Asynchronous Router
The asynchronous router solves the above problems by redesigning the request handling process and introducing an asynchronous processing mechanism. Its core improvements include:
IV. Advantages of the Asynchronous Router
V. Summary
The asynchronous router solves the performance bottleneck problem of the traditional synchronous router in high - concurrency scenarios by introducing an asynchronous processing mechanism. It not only improves the system's concurrency ability and resource utilization but also achieves isolation of downstream services through the queue mechanism, enhancing the system's stability and adaptability. In the federated scenarios where multiple downstream services need to be processed, the asynchronous router is a more efficient and reliable solution.
PRs:
HDFS-17543. [ARR] AsyncUtil makes asynchronous code more concise and easier.
HADOOP-19235. IPC client uses CompletableFuture to support asynchronous operations.
HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony.
HDFS-17545. [ARR] router async rpc client.
HDFS-17594. [ARR] RouterCacheAdmin supports asynchronous rpc.
HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc.
HDFS-17595. [ARR] ErasureCoding supports asynchronous rpc.
HDFS-17601. [ARR] RouterRpcServer supports asynchronous rpc.
HDFS-17596. [ARR] RouterStoragePolicy supports asynchronous rpc.
HDFS-17656. [ARR] RouterNamenodeProtocol and RouterUserProtocol supports asynchronous rpc.
HDFS-17659. [ARR]Router Quota supports asynchronous rpc.
HDFS-17672. [ARR] Move asynchronous related classes to the async package.
HADOOP-19361. RPC DeferredMetrics bugfix.
HDFS-17640.[ARR] RouterClientProtocol supports asynchronous rpc.
HDFS-17650. [ARR] The router server-side rpc protocol PB supports asynchrony.
HDFS-17651.[ARR] Async handler executor isolation.
HDFS-17715. [ARR] Add documentation for asynchronous router.
How was this patch tested?
Async Router RPC: single nameservice performance testing report
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?