Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-14738:Reading and writing HDFS data through HDFS nfs3 server often getting stuck #1300

Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from

Conversation

alexkingli
Copy link

If the user group does not exist, there will be a lot of invoking the updateMapIncr . And it is the synchronized methods,it really affects performance.

@alexkingli alexkingli changed the title update updateMapIncr to avoid loading information form os every time HDFS-14738:Reading and writing HDFS data through HDFS nfs3 server often getting stuck Aug 15, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 45 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1066 trunk passed
+1 compile 1070 trunk passed
+1 checkstyle 40 trunk passed
+1 mvnsite 79 trunk passed
+1 shadedclient 837 branch has no errors when building and testing our client artifacts.
+1 javadoc 60 trunk passed
0 spotbugs 123 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 120 trunk passed
_ Patch Compile Tests _
+1 mvninstall 48 the patch passed
+1 compile 1091 the patch passed
+1 javac 1091 the patch passed
+1 checkstyle 40 the patch passed
+1 mvnsite 81 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 648 patch has no errors when building and testing our client artifacts.
+1 javadoc 63 the patch passed
+1 findbugs 134 the patch passed
_ Other Tests _
-1 unit 507 hadoop-common in the patch failed.
+1 asflicense 40 The patch does not generate ASF License warnings.
6024
Reason Tests
Failed junit tests hadoop.security.TestShellBasedIdMapping
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/artifact/out/Dockerfile
GITHUB PR #1300
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 7e2dab2da4c1 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 9b8359b
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/testReport/
Max. process+thread count 1417 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 551 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 1 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1133 trunk passed
+1 compile 1049 trunk passed
+1 checkstyle 36 trunk passed
+1 mvnsite 72 trunk passed
+1 shadedclient 735 branch has no errors when building and testing our client artifacts.
+1 javadoc 55 trunk passed
0 spotbugs 111 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 110 trunk passed
_ Patch Compile Tests _
+1 mvninstall 53 the patch passed
+1 compile 1020 the patch passed
+1 javac 1020 the patch passed
+1 checkstyle 35 the patch passed
+1 mvnsite 64 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 624 patch has no errors when building and testing our client artifacts.
+1 javadoc 55 the patch passed
+1 findbugs 123 the patch passed
_ Other Tests _
-1 unit 512 hadoop-common in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
6325
Reason Tests
Failed junit tests hadoop.security.TestShellBasedIdMapping
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/artifact/out/Dockerfile
GITHUB PR #1300
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 3ea77b56f5e7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 6244502
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/testReport/
Max. process+thread count 1380 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 41 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1125 trunk passed
+1 compile 1239 trunk passed
+1 checkstyle 39 trunk passed
+1 mvnsite 75 trunk passed
+1 shadedclient 765 branch has no errors when building and testing our client artifacts.
+1 javadoc 60 trunk passed
0 spotbugs 120 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 118 trunk passed
_ Patch Compile Tests _
+1 mvninstall 43 the patch passed
+1 compile 1050 the patch passed
+1 javac 1050 the patch passed
+1 checkstyle 39 the patch passed
+1 mvnsite 72 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 625 patch has no errors when building and testing our client artifacts.
+1 javadoc 56 the patch passed
+1 findbugs 122 the patch passed
_ Other Tests _
-1 unit 544 hadoop-common in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
6111
Reason Tests
Failed junit tests hadoop.security.TestShellBasedIdMapping
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/Dockerfile
GITHUB PR #1300
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 15eb07c9efaf 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ee7c261
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/testReport/
Max. process+thread count 1424 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 57 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1371 trunk passed
+1 compile 1258 trunk passed
+1 checkstyle 51 trunk passed
+1 mvnsite 94 trunk passed
+1 shadedclient 981 branch has no errors when building and testing our client artifacts.
+1 javadoc 75 trunk passed
0 spotbugs 150 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 148 trunk passed
_ Patch Compile Tests _
+1 mvninstall 54 the patch passed
+1 compile 1172 the patch passed
+1 javac 1172 the patch passed
+1 checkstyle 50 the patch passed
+1 mvnsite 82 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 702 patch has no errors when building and testing our client artifacts.
+1 javadoc 71 the patch passed
+1 findbugs 130 the patch passed
_ Other Tests _
-1 unit 532 hadoop-common in the patch failed.
+1 asflicense 47 The patch does not generate ASF License warnings.
6961
Reason Tests
Failed junit tests hadoop.security.TestShellBasedIdMapping
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/artifact/out/Dockerfile
GITHUB PR #1300
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 3a08b1fd0c60 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3329257
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/testReport/
Max. process+thread count 1513 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 97 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1329 trunk passed
+1 compile 1233 trunk passed
+1 checkstyle 47 trunk passed
+1 mvnsite 92 trunk passed
+1 shadedclient 927 branch has no errors when building and testing our client artifacts.
+1 javadoc 60 trunk passed
0 spotbugs 141 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 139 trunk passed
_ Patch Compile Tests _
+1 mvninstall 49 the patch passed
+1 compile 1148 the patch passed
+1 javac 1148 the patch passed
+1 checkstyle 45 the patch passed
+1 mvnsite 77 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 774 patch has no errors when building and testing our client artifacts.
+1 javadoc 72 the patch passed
+1 findbugs 133 the patch passed
_ Other Tests _
-1 unit 571 hadoop-common in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
6900
Reason Tests
Failed junit tests hadoop.security.TestShellBasedIdMapping
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/artifact/out/Dockerfile
GITHUB PR #1300
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux eb6d25b5071b 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 915cbc9
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/testReport/
Max. process+thread count 1378 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 20m 30s trunk passed
+1 💚 compile 21m 31s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 compile 18m 35s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 checkstyle 0m 48s trunk passed
+1 💚 mvnsite 1m 28s trunk passed
+1 💚 shadedclient 16m 51s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 39s hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚 javadoc 0m 59s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗 spotbugs 2m 13s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 11s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 55s the patch passed
+1 💚 compile 20m 4s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javac 20m 4s the patch passed
+1 💚 compile 17m 50s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 javac 17m 50s the patch passed
+1 💚 checkstyle 0m 45s the patch passed
+1 💚 mvnsite 1m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 6s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 40s hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚 javadoc 0m 57s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 findbugs 2m 29s the patch passed
_ Other Tests _
-1 ❌ unit 9m 26s hadoop-common in the patch passed.
+1 💚 asflicense 0m 50s The patch does not generate ASF License warnings.
155m 35s
Reason Tests
Failed junit tests hadoop.security.TestShellBasedIdMapping
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/Dockerfile
GITHUB PR #1300
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 139168d963b2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e756fe3
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
unit https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/testReport/
Max. process+thread count 3449 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/console
versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants