HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat… #2135

touchida · 2020-07-12T06:57:05Z

…anode through insecure Namenode.

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

…anode through insecure Namenode.

hadoop-yetus · 2020-07-12T10:07:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	20m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 59s	trunk passed
+1 💚	compile	1m 16s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	1m 6s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 48s	trunk passed
+1 💚	mvnsite	1m 15s	trunk passed
+1 💚	shadedclient	17m 9s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 32s	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 40s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	3m 3s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 1s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 9s	the patch passed
+1 💚	compile	1m 12s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	1m 12s	the patch passed
+1 💚	compile	1m 4s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	1m 4s	the patch passed
+1 💚	checkstyle	0m 42s	the patch passed
+1 💚	mvnsite	1m 8s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 14s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 28s	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 38s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	3m 3s	the patch passed
		_ Other Tests _
-1 ❌	unit	91m 40s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 34s	The patch does not generate ASF License warnings.
		186m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestGetFileChecksum

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2135/1/artifact/out/Dockerfile
GITHUB PR	#2135
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ad29aa6754ad 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `84b74b3`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2135/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2135/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2135/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2135/1/testReport/
Max. process+thread count	3460 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2135/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

sunchao · 2020-07-15T21:51:19Z

@touchida thanks for the PR. What is the implication of this when both NN and DN are secure, and some insecure client try the following:

curl -i "http://<SECURE_DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&namenoderpcaddress=<SECURE_NAMENODE>&offset=0"

will it still work and cause security breach since secure DN can launch a DFS client talking to the secure NN?

touchida · 2020-07-21T12:28:51Z

@sunchao Thanks for your comment!

curl -i "http://<SECURE_DATANODE>:/webhdfs/v1/?op=OPEN&namenoderpcaddress=<SECURE_NAMENODE>&offset=0"

No, it won't work.
It will result in AccessControlException with 403 response code, as follows.

$ curl -i "http://<SECURE_DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&namenoderpcaddress=<SECURE_NAMENODE>&offset=0"
HTTP/1.1 403 Forbidden
(omitted)
{"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"DestHost:destPort <SECURE_NAMENODE>:<PORT> , LocalHost:localPort <SECURE_DATANODE>:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]"}}

The corresponding Datanode log is as follows:

2020-07-21 09:16:02,559 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2020-07-21 09:16:02,577 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2020-07-21 09:16:02,578 INFO org.apache.hadoop.io.retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort <SECURE_NAMNODE>:<PORT> , LocalHost:localPort <SECURE_DATANODE>:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over <SECURE_NAMENODE>:<PORT> after 1 failover attempts. Trying to failover after sleeping for 1224ms.
(omitted)
2020-07-21 09:18:40,881 INFO org.apache.hadoop.io.retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort <SECURE_NAMNODE>:<PORT> , LocalHost:localPort <SECURE_DATANODE>:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over <SECURE_NAMENODE>:<PORT> after 14 failover attempts. Trying to failover after sleeping for 20346ms.
(omitted)
2020-07-21 09:19:01,243 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

This is because in the absence of delegation tokens, org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler#channelRead0 will create insecure DFSClient, which cannot talk to secure Namenode.

https://github.com/apache/hadoop/blob/da0006f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java#L261

sunchao

Thanks @touchida ! This LGTM.

touchida · 2020-07-27T15:56:23Z

@sunchao Thanks for your review!

The failed tests are not related to the PR since they don't use WebHDFS.
Most of them have already been reported as flaky.

org.apache.hadoop.hdfs.TestGetFileChecksum.testGetFileChecksum (HDFS-15461)
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead

java.net.BindException: Problem binding to [localhost:18020] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException

org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics.testRetryCacheMetrics (HDFS-15458)
org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier.testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace (HDFS-15456)

Ready to be merged.

sunchao · 2020-07-27T16:55:19Z

Merged. Thanks @touchida !

…anode through insecure Namenode. (apache#2135)

…anode through insecure Namenode. (apache#2135) Change-Id: Ib9ab9f6f5a37c8f5a6ff204f19366305455b2637

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat…

9c81a16

…anode through insecure Namenode.

sunchao approved these changes Jul 24, 2020

View reviewed changes

sunchao merged commit 026dce5 into apache:trunk Jul 27, 2020

bilaharith pushed a commit to bilaharith/hadoop that referenced this pull request Jul 29, 2020

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat…

00017af

…anode through insecure Namenode. (apache#2135)

vitojeng mentioned this pull request Dec 7, 2022

HDFS-15465: Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode djydata/hadoop#1

Open

vitojeng pushed a commit to djydata/hadoop that referenced this pull request Dec 12, 2022

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat…

0ac1b6f

…anode through insecure Namenode. (apache#2135)

vitojeng mentioned this pull request Dec 12, 2022

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat… djydata/hadoop#3

Open

jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat…

fb1ca35

…anode through insecure Namenode. (apache#2135) Change-Id: Ib9ab9f6f5a37c8f5a6ff204f19366305455b2637

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat… #2135

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat… #2135

Uh oh!

touchida commented Jul 12, 2020

Uh oh!

hadoop-yetus commented Jul 12, 2020

Uh oh!

sunchao commented Jul 15, 2020

Uh oh!

touchida commented Jul 21, 2020

Uh oh!

sunchao left a comment

Uh oh!

touchida commented Jul 27, 2020

Uh oh!

sunchao commented Jul 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat… #2135

HDFS-15465. Support WebHDFS accesses to the data stored in secure Dat… #2135

Uh oh!

Conversation

touchida commented Jul 12, 2020

NOTICE

Uh oh!

hadoop-yetus commented Jul 12, 2020

Uh oh!

sunchao commented Jul 15, 2020

Uh oh!

touchida commented Jul 21, 2020

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

touchida commented Jul 27, 2020

Uh oh!

sunchao commented Jul 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants