Skip to content

HDFS-15818 Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. #2679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 9, 2021
Merged

HDFS-15818 Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. #2679

merged 3 commits into from
Feb 9, 2021

Conversation

@LeonGao91
Copy link
Contributor

@LeonGao91 LeonGao91 commented Feb 4, 2021

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@hadoop-yetus
Copy link

hadoop-yetus commented Feb 4, 2021

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 34m 12s trunk passed
+1 💚 compile 1m 20s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 compile 1m 13s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 checkstyle 0m 59s trunk passed
+1 💚 mvnsite 1m 23s trunk passed
+1 💚 shadedclient 16m 5s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 54s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 1m 32s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗 spotbugs 3m 19s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 15s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 15s the patch passed
+1 💚 compile 1m 20s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javac 1m 20s the patch passed
+1 💚 compile 1m 8s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 javac 1m 8s the patch passed
-0 ⚠️ checkstyle 0m 56s /diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 41 unchanged - 0 fixed = 43 total (was 41)
+1 💚 mvnsite 1m 16s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 58s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 48s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 1m 21s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 findbugs 3m 7s the patch passed
_ Other Tests _
-1 ❌ unit 195m 42s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
284m 43s
Reason Tests
Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/1/artifact/out/Dockerfile
GITHUB PR #2679
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 4af8bbd0fc69 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 15a1f7a
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/1/testReport/
Max. process+thread count 3118 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/1/console
versions git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@sodonnel
Copy link
Contributor

sodonnel commented Feb 5, 2021

Thanks for fixing this issue. The changes look good to me. Can you fix the checkstyle warnings please?

@hadoop-yetus
Copy link

hadoop-yetus commented Feb 6, 2021

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 51s trunk passed
+1 💚 compile 1m 30s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 compile 1m 21s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 checkstyle 1m 2s trunk passed
+1 💚 mvnsite 1m 22s trunk passed
+1 💚 shadedclient 15m 35s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 55s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 1m 25s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗 spotbugs 3m 6s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 3s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 13s the patch passed
+1 💚 compile 1m 15s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javac 1m 15s the patch passed
+1 💚 compile 1m 4s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 javac 1m 4s the patch passed
+1 💚 checkstyle 0m 54s the patch passed
+1 💚 mvnsite 1m 15s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 12m 41s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 50s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 1m 25s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 findbugs 3m 5s the patch passed
_ Other Tests _
-1 ❌ unit 245m 43s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 43s The patch does not generate ASF License warnings.
332m 38s
Reason Tests
Failed junit tests hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
hadoop.hdfs.TestEncryptionZones
hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
hadoop.hdfs.TestInjectionForSimulatedStorage
hadoop.hdfs.server.balancer.TestBalancerRPCDelay
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile
hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped
hadoop.hdfs.server.mover.TestMover
hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS
hadoop.hdfs.TestErasureCodingExerciseAPIs
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode
hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/2/artifact/out/Dockerfile
GITHUB PR #2679
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 4d49cc1d1168 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c22c77a
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/2/testReport/
Max. process+thread count 3071 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/2/console
versions git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Feb 7, 2021

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 5s trunk passed
+1 💚 compile 1m 19s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 compile 1m 10s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 checkstyle 1m 3s trunk passed
+1 💚 mvnsite 1m 21s trunk passed
+1 💚 shadedclient 15m 25s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 53s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 1m 27s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗 spotbugs 3m 6s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 4s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 12s the patch passed
+1 💚 compile 1m 11s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javac 1m 11s the patch passed
+1 💚 compile 1m 9s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 javac 1m 9s the patch passed
+1 💚 checkstyle 0m 53s the patch passed
+1 💚 mvnsite 1m 9s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 12m 44s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 48s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 1m 24s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 findbugs 3m 4s the patch passed
_ Other Tests _
-1 ❌ unit 190m 32s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
274m 52s
Reason Tests
Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/3/artifact/out/Dockerfile
GITHUB PR #2679
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f3f6059289fe 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 19ae0fa
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/3/testReport/
Max. process+thread count 3248 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2679/3/console
versions git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@LeonGao91
Copy link
Contributor Author

LeonGao91 commented Feb 7, 2021

@sodonnel Thanks for the review! Fixed checkstyle

sodonnel
sodonnel approved these changes Feb 9, 2021
View reviewed changes
Copy link
Contributor

@sodonnel sodonnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me. I will commit to trunk shortly.

@sodonnel sodonnel merged commit 9434c1e into apache:trunk Feb 9, 2021
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Sep 1, 2021
@LeonGao91 @amahussein
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. …
0c20e27 
…Contributed by Leon Gao (apache#2679)

(cherry picked from commit 9434c1e)
Hexiaoqiao pushed a commit that referenced this pull request Sep 11, 2021
@amahussein @LeonGao91
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various…
318bc5f 
… FsDatasetImpl methods should use datanode readlock. (#3200)

* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1e)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
asfgit pushed a commit that referenced this pull request Sep 14, 2021
@LeonGao91 @Hexiaoqiao
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. …
3c03ade 
…Contributed by Leon Gao (#2679)

(cherry picked from commit 9434c1e)
asfgit pushed a commit that referenced this pull request Sep 14, 2021
@LeonGao91 @Hexiaoqiao
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. …
38973a5 
…Contributed by Leon Gao (#2679)

(cherry picked from commit 9434c1e)
basapuram-kumar pushed a commit to acceldata-io/hadoop that referenced this pull request Jun 25, 2024
@LeonGao91 @basapuram-kumar
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. …
d9fa933 
…Contributed by Leon Gao (apache#2679)

(cherry picked from commit 9434c1e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sodonnel sodonnel sodonnel approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@LeonGao91 @hadoop-yetus @sodonnel