Skip to content

Conversation

@ZanderXu
Copy link
Contributor

NameNode supports dynamically change ErasureCodingPolicy, so these ErasureCodingPolicies should be protected by one lock, the current implementation uses the global lock.

ErasureCodingPolicy mainly involves directory tree and edits logs, such as:

  • getErasureCodingPolicy(String src)
  • setErasureCodingPolicy(String src, String ecPolicyName)
  • addErasureCodingPolicies(ErasureCodingPolicy[] policies)
  • disableErasureCodingPolicy(String ecPolicyName)
  • enableErasureCodingPolicy(String ecPolicyName)

So we can use the FSLock to make these operations thread safe.

Another reason why we use the FSLock to protect ErasureCodingPolicy related operations is that we use FSLock to make edit write operations thread safe. 

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ HDFS-17384 Compile Tests _
+1 💚 mvninstall 32m 5s HDFS-17384 passed
+1 💚 compile 0m 45s HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 0m 41s HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 41s HDFS-17384 passed
+1 💚 mvnsite 0m 44s HDFS-17384 passed
+1 💚 javadoc 0m 40s HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 5s HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 42s HDFS-17384 passed
+1 💚 shadedclient 20m 19s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 36s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 0m 36s the patch passed
+1 💚 compile 0m 35s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 35s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 30s the patch passed
+1 💚 mvnsite 0m 36s the patch passed
+1 💚 javadoc 0m 32s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 57s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 1m 41s the patch passed
+1 💚 shadedclient 20m 26s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 207m 6s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
301m 58s
Reason Tests
Failed junit tests hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.server.datanode.TestLargeBlockReport
hadoop.hdfs.tools.TestDFSAdmin
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6579/3/artifact/out/Dockerfile
GITHUB PR #6579
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3a65e8ced70c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HDFS-17384 / 1ba05cb
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6579/3/testReport/
Max. process+thread count 5351 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6579/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@ferhui ferhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clear to me. +1

@haiyang1987
Copy link
Contributor

LGTM. +1.

@ZanderXu
Copy link
Contributor Author

ZanderXu commented Mar 7, 2024

Thanks @ferhui and @haiyang1987 for your review. Merged

@ZanderXu ZanderXu merged commit fde6437 into apache:HDFS-17384 Mar 7, 2024
ZanderXu added a commit to ZanderXu/hadoop that referenced this pull request Mar 25, 2024
ZanderXu added a commit that referenced this pull request Jan 7, 2025
YanivKunda pushed a commit to YanivKunda/hadoop that referenced this pull request Mar 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants