HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony. #6870

KeeProMise · 2024-06-06T15:37:35Z

Description of PR

Please see : https://issues.apache.org/jira/browse/HDFS-17544

NOTE:
This is a sub-pull request (PR) related to HDFS-17531(Asynchronous router RPC). For more details or context, please refer to the main issue HDFS-17531
More detailed documentation: HDFS-17531 Router asynchronous rpc implementation.pdf and Aynchronous router.pdf

How was this patch tested?

new UTs:

1.TestAsyncRpcProtocolPBUtil
2.TestRouterClientSideTranslatorPB

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hfutatzhanghb · 2024-06-07T02:24:00Z

@KeeProMise Hi, sir. The code in this PR contains HDFS-17543's change. Can we seperate them for better reviewing.

KeeProMise · 2024-06-07T02:31:17Z

@KeeProMise Hi, sir. The code in this PR contains HDFS-17543's change. Can we seperate them for better reviewing.

@hfutatzhanghb Hi, It does include HDFS-17543. The main reason is that this PR needs to use AsyncUtil of HDFS-17543. We should first review and merge HDFS-17543 and then review this PR. Before HDFS-17543 is merged, this PR is only used as an example of subsequent use of AsyncUtil.

KeeProMise · 2024-07-09T09:01:53Z

@goiri @simbadzina @Hexiaoqiao @sjlee @ayushtkn
Hi, if you have time, please help to take a look at this PR, thanks!

KeeProMise · 2024-07-09T09:43:29Z

Modifications related to ipc.client, please see to HADOOP-19235 .#6888

hadoop-yetus · 2024-07-25T03:06:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 20s		#6870 does not apply to HDFS-17531. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#6870
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/13/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-07-25T12:30:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ HDFS-17531 Compile Tests _
+0 🆗	mvndep	13m 59s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 41s		HDFS-17531 passed
+1 💚	compile	2m 55s		HDFS-17531 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	2m 49s		HDFS-17531 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	0m 44s		HDFS-17531 passed
+1 💚	mvnsite	0m 56s		HDFS-17531 passed
+1 💚	javadoc	0m 56s		HDFS-17531 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 50s		HDFS-17531 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	2m 12s		HDFS-17531 passed
+1 💚	shadedclient	20m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 47s		the patch passed
+1 💚	compile	2m 49s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	2m 49s		the patch passed
+1 💚	compile	2m 41s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	2m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 36s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 2 new + 22 unchanged - 1 fixed = 24 total (was 23)
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 41s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 42s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	2m 13s		the patch passed
+1 💚	shadedclient	20m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 44s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	27m 23s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		135m 25s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.server.federation.router.TestRouterRpc
	hadoop.hdfs.server.federation.router.TestRouterMultiRack

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/16/artifact/out/Dockerfile
GITHUB PR	#6870
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux aff8b889771a 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17531 / `c311ab6`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/16/testReport/
Max. process+thread count	3709 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/16/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-07-25T18:08:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ HDFS-17531 Compile Tests _
+0 🆗	mvndep	14m 23s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 11s		HDFS-17531 passed
+1 💚	compile	2m 50s		HDFS-17531 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	2m 46s		HDFS-17531 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	0m 44s		HDFS-17531 passed
+1 💚	mvnsite	1m 2s		HDFS-17531 passed
+1 💚	javadoc	0m 54s		HDFS-17531 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 48s		HDFS-17531 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	2m 10s		HDFS-17531 passed
+1 💚	shadedclient	20m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	2m 50s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	2m 50s		the patch passed
+1 💚	compile	2m 39s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	2m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		hadoop-hdfs-project: The patch generated 0 new + 22 unchanged - 1 fixed = 22 total (was 23)
+1 💚	mvnsite	0m 51s		the patch passed
+1 💚	javadoc	0m 41s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 43s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	2m 15s		the patch passed
+1 💚	shadedclient	20m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 45s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	26m 56s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		128m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/17/artifact/out/Dockerfile
GITHUB PR	#6870
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux fe27c75d2ac5 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17531 / `21a1b1a`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/17/testReport/
Max. process+thread count	3684 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6870/17/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

KeeProMise · 2024-07-26T01:58:32Z

@goiri @simbadzina @Hexiaoqiao @sjlee @ayushtkn @haiyang1987 @ZanderXu
Hi, if you have time, please help to take a look at this PR, thanks!

Hexiaoqiao

LGTM

KeeProMise · 2024-08-01T07:26:46Z

LGTM

Hi, if no comments here, please help to merge it, thanks!

Hexiaoqiao · 2024-08-02T10:22:43Z

Committed to HDFS-17531, Thanks @KeeProMise !

KeeProMise · 2024-08-02T13:02:46Z

Committed to HDFS-17531, Thanks @KeeProMise !

@Hexiaoqiao Thank you for your review and merge!

…ny. (apache#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

github-actions bot added HDFS trunk Common RBF labels Jun 6, 2024

KeeProMise changed the title ~~Hdfs-17544. [ARR] The router client rpc protocol supports asynchrony.~~ HDFS-17544. [ARR] The router client rpc protocol supports asynchrony. Jun 6, 2024

KeeProMise closed this Jun 6, 2024

KeeProMise reopened this Jun 6, 2024

KeeProMise force-pushed the HDFS-17544 branch from 079e1c9 to 646ed92 Compare July 3, 2024 17:42

KeeProMise changed the title ~~HDFS-17544. [ARR] The router client rpc protocol supports asynchrony.~~ HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony. Jul 9, 2024

KeeProMise force-pushed the HDFS-17544 branch from 2798671 to 08f1e96 Compare July 25, 2024 03:23

rebase

c311ab6

KeeProMise force-pushed the HDFS-17544 branch from 5ecbf50 to c311ab6 Compare July 25, 2024 10:14

fix ut & checkstyle

21a1b1a

Hexiaoqiao approved these changes Jul 28, 2024

View reviewed changes

Hexiaoqiao merged commit 4e0b405 into apache:HDFS-17531 Aug 2, 2024
4 checks passed

KeeProMise added a commit to KeeProMise/hadoop that referenced this pull request Sep 9, 2024

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchro…

7701931

…ny. (apache#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

Hexiaoqiao pushed a commit to Hexiaoqiao/hadoop that referenced this pull request Sep 12, 2024

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchro…

23deccf

…ny. (apache#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

Hexiaoqiao pushed a commit to Hexiaoqiao/hadoop that referenced this pull request Sep 12, 2024

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchro…

a591902

…ny. (apache#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

KeeProMise added a commit to KeeProMise/hadoop that referenced this pull request Nov 21, 2024

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchro…

00564f0

…ny. (apache#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

KeeProMise added a commit to KeeProMise/hadoop that referenced this pull request Nov 21, 2024

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchro…

9217317

…ny. (apache#6870). Contributed by Jian Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony. #6870

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony. #6870

KeeProMise commented Jun 6, 2024 •

edited

Loading

hfutatzhanghb commented Jun 7, 2024

KeeProMise commented Jun 7, 2024

KeeProMise commented Jul 9, 2024

KeeProMise commented Jul 9, 2024 •

edited

Loading

hadoop-yetus commented Jul 25, 2024

hadoop-yetus commented Jul 25, 2024

hadoop-yetus commented Jul 25, 2024

KeeProMise commented Jul 26, 2024 •

edited

Loading

Hexiaoqiao left a comment

KeeProMise commented Aug 1, 2024

Hexiaoqiao commented Aug 2, 2024

KeeProMise commented Aug 2, 2024

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony. #6870

HDFS-17544. [ARR] The router client rpc protocol PB supports asynchrony. #6870

Conversation

KeeProMise commented Jun 6, 2024 • edited Loading

Description of PR

How was this patch tested?

For code changes:

hfutatzhanghb commented Jun 7, 2024

KeeProMise commented Jun 7, 2024

KeeProMise commented Jul 9, 2024

KeeProMise commented Jul 9, 2024 • edited Loading

hadoop-yetus commented Jul 25, 2024

hadoop-yetus commented Jul 25, 2024

hadoop-yetus commented Jul 25, 2024

KeeProMise commented Jul 26, 2024 • edited Loading

Hexiaoqiao left a comment

Choose a reason for hiding this comment

KeeProMise commented Aug 1, 2024

Hexiaoqiao commented Aug 2, 2024

KeeProMise commented Aug 2, 2024

KeeProMise commented Jun 6, 2024 •

edited

Loading

KeeProMise commented Jul 9, 2024 •

edited

Loading

KeeProMise commented Jul 26, 2024 •

edited

Loading