HDFS-17588. RBF: RouterObserverReadProxyProvider should perform an msync before executing the first read. #6956

fuchaohong · 2024-07-23T05:47:15Z

When using RouterObserverReadProxyProvider to initiate the first RPC request, the router routes this RPC to the active namenode and updates the stateid of the corresponding nameservice. However, the stateid of other nameservices is not updated. Clients should first perform msync to update the stateid of all nameservices with enabled Observers before executing the first read.

…xecuting the first read.

hadoop-yetus · 2024-07-23T09:18:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 48s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 16s		trunk passed
+1 💚	compile	5m 28s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	5m 21s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	1m 25s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 28s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 17s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 56s		trunk passed
+1 💚	shadedclient	34m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	5m 23s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	5m 23s		the patch passed
+1 💚	compile	5m 10s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	5m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 20s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0)
+1 💚	mvnsite	1m 27s		the patch passed
+1 💚	javadoc	1m 10s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 2s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	4m 35s		the patch passed
-1 ❌	shadedclient	49m 31s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	16m 44s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	0m 28s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		210m 6s

Reason	Tests
Failed junit tests	hadoop.hdfs.web.TestWebHDFSOAuth2

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/1/artifact/out/Dockerfile
GITHUB PR	#6956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 336bba1ea4d8 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1fea71b`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/1/testReport/
Max. process+thread count	552 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-07-23T13:08:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 13s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 56s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 50s		trunk passed
+1 💚	compile	5m 29s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	5m 24s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	1m 23s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 29s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 19s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 55s		trunk passed
+1 💚	shadedclient	33m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	5m 20s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	5m 20s		the patch passed
+1 💚	compile	5m 13s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	5m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 14s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0)
+1 💚	mvnsite	1m 27s		the patch passed
+1 💚	javadoc	1m 8s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 2s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 59s		the patch passed
+1 💚	shadedclient	34m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	30m 17s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		205m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpc

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/2/artifact/out/Dockerfile
GITHUB PR	#6956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 269f36d8d86b 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `76eec42`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/2/testReport/
Max. process+thread count	3876 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-07-24T12:38:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 28s		trunk passed
+1 💚	compile	5m 29s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	5m 22s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	1m 27s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 30s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 19s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 56s		trunk passed
+1 💚	shadedclient	34m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 25s		the patch passed
+1 💚	compile	5m 22s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	5m 22s		the patch passed
+1 💚	compile	5m 12s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	5m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 13s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0)
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	1m 8s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 4s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	4m 0s		the patch passed
+1 💚	shadedclient	34m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 28s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	32m 25s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		196m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/3/artifact/out/Dockerfile
GITHUB PR	#6956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 179d10dea9e3 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c9b3dfd`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/3/testReport/
Max. process+thread count	4396 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

fuchaohong · 2024-07-25T01:09:51Z

@simbadzina Master, please help me review this PR.

simbadzina

Hi. Sorry for taking so long to review. I haven't had much bandwidth for the last few months.
A number of tests in TestObserverWithRouter start to fail due to the extra msync. The counts of operations needs to be adjusted for these tests.
I don't know why the CI didn't run this class: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/3/testReport/org.apache.hadoop.hdfs.server.federation.router/

…o HDFS-17588

fuchaohong · 2024-11-07T07:30:44Z

@simbadzina testRestartingNamenodeWithStateIDContextDisabled failed due to stateID=0，HDFS-17514 allows passing stateID=0, I wonder if this can be fixed？

hadoop-yetus · 2024-11-07T10:25:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 49s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 16s		trunk passed
+1 💚	compile	5m 25s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	5m 15s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	1m 25s		trunk passed
+1 💚	mvnsite	1m 41s		trunk passed
+1 💚	javadoc	1m 31s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 19s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	3m 56s		trunk passed
+1 💚	shadedclient	35m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	5m 16s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	5m 16s		the patch passed
+1 💚	compile	5m 7s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	5m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 14s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0)
+1 💚	mvnsite	1m 25s		the patch passed
+1 💚	javadoc	1m 12s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 0s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	3m 58s		the patch passed
+1 💚	shadedclient	35m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 35s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	32m 41s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		198m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/4/artifact/out/Dockerfile
GITHUB PR	#6956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e1e7d2e33e89 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `92e5e19`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/4/testReport/
Max. process+thread count	3770 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6956/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

RBF: RouterObserverReadProxyProvider should perform an msync before e…

1fea71b

…xecuting the first read.

github-actions bot added HDFS trunk RBF labels Jul 23, 2024

triggle compile

76eec42

Update TestObserverWithRouter.java

c9b3dfd

simbadzina requested changes Oct 30, 2024

View reviewed changes

fuchaohong added 2 commits November 7, 2024 14:55

Fix UT.

9fba56d

Merge branch 'HDFS-17588' of https://github.com/fuchaohong/hadoop int…

92e5e19

…o HDFS-17588

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17588. RBF: RouterObserverReadProxyProvider should perform an msync before executing the first read. #6956

HDFS-17588. RBF: RouterObserverReadProxyProvider should perform an msync before executing the first read. #6956

fuchaohong commented Jul 23, 2024 •

edited

Loading

hadoop-yetus commented Jul 23, 2024

hadoop-yetus commented Jul 23, 2024

hadoop-yetus commented Jul 24, 2024

fuchaohong commented Jul 25, 2024

simbadzina left a comment •

edited

Loading

fuchaohong commented Nov 7, 2024

hadoop-yetus commented Nov 7, 2024

HDFS-17588. RBF: RouterObserverReadProxyProvider should perform an msync before executing the first read. #6956

Are you sure you want to change the base?

HDFS-17588. RBF: RouterObserverReadProxyProvider should perform an msync before executing the first read. #6956

Conversation

fuchaohong commented Jul 23, 2024 • edited Loading

hadoop-yetus commented Jul 23, 2024

hadoop-yetus commented Jul 23, 2024

hadoop-yetus commented Jul 24, 2024

fuchaohong commented Jul 25, 2024

simbadzina left a comment • edited Loading

Choose a reason for hiding this comment

fuchaohong commented Nov 7, 2024

hadoop-yetus commented Nov 7, 2024

fuchaohong commented Jul 23, 2024 •

edited

Loading

simbadzina left a comment •

edited

Loading