Merge branch 'master' into HDDS-10239-container-reconciliation

COMMITS: 360fea5 HDDS-11494. Improve the duration option of freon ombg (apache#7246) 10d3b21 HDDS-11504. Update Ratis to 3.1.1. (apache#7257) ce46297 HDDS-11162. Improve Disk Usage page UI (apache#7214) c91f1c7 HDDS-11491. Avoid sharing clientId among deleting services (apache#7250) b0943d5 HDDS-11501. Improve logging in XceiverServerRatis (apache#7252) 55925ab HDDS-11502. Class path contains multiple SLF4J providers (apache#7255) d0ad836 HDDS-11472. Avoid recreating external access authorizer on OM state reload (apache#7238) 254db9e HDDS-11500. RootCARotationManager cancelling wrong task in notifyStatusChanged (apache#7251) 1e6e4b3 HDDS-11499. Remove redundant code from ECReconstructionCoordinator. (apache#7248) adb2821 HDDS-11490. Bump rollup to 3.29.5 (apache#7232) 189a9fe HDDS-11484. Validate javadoc in CI (apache#7245) 64a29c6 HDDS-11497. Bump commons-configuration2 to 2.11.0 (apache#7242) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snapshot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForPrefixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyInFailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snaps hot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForP refixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyIn FailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) e0060a8 HDDS-11196. Improve SCM WebUI Display (apache#6960) 22ddfb9 Revert "HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192)" 9f5bf43 HDDS-11457. Internal error on S3 CompleteMultipartUpload if parts are not specified (apache#7195) 10c47a1 HDDS-11459. Bump develocity-maven-extension to 1.22.1 (apache#7201) 50f2563 HDDS-11419. Fix waitForCheckpointDirectoryExist log message (apache#7199) a7d7e37 HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192) 5feb9ea HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch (apache#7188) 703c4d5 HDDS-10984. Tool to restore SCM certificates from RocksDB. (apache#6781) d221065 HDDS-11440. Add a lastTransactionInfo field in SnapshotInfo to check for transactions in flight on the snapshot (apache#7179) e573701 HDDS-11448. Improve documentation in ContainerStateMachine (apache#7183) 0e49f7a HDDS-11449. Remove unnecessary log from client console. (apache#7184) cd251f2 HDDS-11438. Ensure DataInputBuffer is closed in OMPBHelper#convert (apache#7182) 4b47812 HDDS-11389. Incorrect number of deleted containers shown in Recon UI. (apache#7149) 0915f0b HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. (apache#7009) 0f16195 HDDS-11416. refactor ratis submit request avoid code duplicate (apache#7166) 86fe920 HDDS-11376. Improve ReplicationSupervisor to record replication metrics (apache#7140) CONFLICT: Merge conflict in hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandlerWithUnhealthyContainer.java MODIFIED: /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
aswinshakil · Oct 1, 2024 · 2b22308 · 2b22308
2 parents 9765a6a + 360fea5
commit 2b22308
Show file tree

Hide file tree

Showing 440 changed files with 15,001 additions and 5,905 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -218,7 +218,7 @@ jobs:
           distribution: 'temurin'
           java-version: ${{ matrix.java }}
       - name: Compile Ozone using Java ${{ matrix.java }}
-        run: hadoop-ozone/dev-support/checks/build.sh -Pdist -Dskip.npx -Dskip.installnpx -Djavac.version=${{ matrix.java }} ${{ inputs.ratis_args }}
+        run: hadoop-ozone/dev-support/checks/build.sh -Pdist -Dskip.npx -Dskip.installnpx -Dmaven.javadoc.failOnWarnings=${{ matrix.java != 8 }} -Djavac.version=${{ matrix.java }} ${{ inputs.ratis_args }}
         env:
           OZONE_WITH_COVERAGE: false
           DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}

diff --git a/.mvn/extensions.xml b/.mvn/extensions.xml
@@ -24,7 +24,7 @@
   <extension>
     <groupId>com.gradle</groupId>
     <artifactId>develocity-maven-extension</artifactId>
-    <version>1.21.5</version>
+    <version>1.22.1</version>
   </extension>
   <extension>
     <groupId>com.gradle</groupId>

diff --git a/NOTICE.txt b/NOTICE.txt
@@ -1,5 +1,5 @@
 Apache Ozone
-Copyright 2022 The Apache Software Foundation
+Copyright 2024 The Apache Software Foundation
 
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).

diff --git a/dev-support/ci/selective_ci_checks.bats b/dev-support/ci/selective_ci_checks.bats
@@ -429,3 +429,15 @@ load bats-assert/load.bash
   assert_output -p needs-integration-tests=false
   assert_output -p needs-kubernetes-tests=false
 }
+
+@test "properties file in resources" {
+  run dev-support/ci/selective_ci_checks.sh 71b8bdd8becf72d6f7d4e7986895504b8259b3e5
+
+  assert_output -p 'basic-checks=["rat","checkstyle","native"]'
+  assert_output -p needs-build=false
+  assert_output -p needs-compile=false
+  assert_output -p needs-compose-tests=false
+  assert_output -p needs-dependency-check=false
+  assert_output -p needs-integration-tests=true
+  assert_output -p needs-kubernetes-tests=false
+}
diff --git a/dev-support/ci/selective_ci_checks.sh b/dev-support/ci/selective_ci_checks.sh
@@ -373,6 +373,7 @@ function check_needs_checkstyle() {
         "^hadoop-hdds/dev-support/checkstyle"
         "pom.xml"
         "src/..../java"
+        "src/..../resources/.*\.properties"
     )
     local ignore_array=(
         "^hadoop-ozone/dist"

diff --git a/dev-support/pom.xml b/dev-support/pom.xml
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <parent>
+    <artifactId>ozone-main</artifactId>
+    <groupId>org.apache.ozone</groupId>
+    <version>1.5.0-SNAPSHOT</version>
+  </parent>
+  <modelVersion>4.0.0</modelVersion>
+  <artifactId>ozone-dev-support</artifactId>
+  <description>Helper module for sharing resources among projects</description>
+  <name>Apache Ozone Dev Support</name>
+
+  <properties>
+    <failIfNoTests>false</failIfNoTests>
+  </properties>
+  <build>
+    <resources>
+      <resource>
+        <directory>${project.build.directory}/extra-resources</directory>
+        <targetPath>META-INF</targetPath>
+        <includes>
+          <include>LICENSE.txt</include>
+          <include>NOTICE.txt</include>
+        </includes>
+      </resource>
+    </resources>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-site-plugin</artifactId>
+        <configuration>
+          <skip>true</skip>
+        </configuration>
+      </plugin>
+      <!-- copy L&N files to target/extra-resources -->
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-resources-plugin</artifactId>
+        <executions>
+          <execution>
+            <id>copy-resources</id>
+            <phase>validate</phase>
+            <goals>
+              <goal>copy-resources</goal>
+            </goals>
+            <configuration>
+              <outputDirectory>${project.build.directory}/extra-resources</outputDirectory>
+              <resources>
+                <resource>
+                  <directory>../</directory>
+                  <includes>
+                    <include>LICENSE.txt</include>
+                    <include>NOTICE.txt</include>
+                  </includes>
+                </resource>
+              </resources>
+            </configuration>
+          </execution>
+        </executions>
+      </plugin>
+      <!-- add entries for L&N files to remote-resources.xml in jar file -->
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-remote-resources-plugin</artifactId>
+        <executions>
+          <execution>
+            <phase>process-resources</phase>
+            <goals>
+              <goal>bundle</goal>
+            </goals>
+          </execution>
+        </executions>
+        <configuration>
+          <resourcesDirectory>${project.build.outputDirectory}</resourcesDirectory>
+          <includes>
+            <include>META-INF/LICENSE.txt</include>
+            <include>META-INF/NOTICE.txt</include>
+          </includes>
+        </configuration>
+      </plugin>
+    </plugins>
+  </build>
+</project>
diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java
@@ -247,28 +247,49 @@ public enum ChecksumCombineMode {
       tags = ConfigTag.CLIENT)
   private String fsDefaultBucketLayout = "FILE_SYSTEM_OPTIMIZED";
 
+  // ozone.client.hbase.enhancements.allowed
+  @Config(key = "hbase.enhancements.allowed",
+      defaultValue = "false",
+      description = "When set to false, client-side HBase enhancement-related Ozone (experimental) features " +
+          "are disabled (not allowed to be enabled) regardless of whether those configs are set.\n" +
+          "\n" +
+          "Here is the list of configs and values overridden when this config is set to false:\n" +
+          "1. ozone.fs.hsync.enabled = false\n" +
+          "2. ozone.client.incremental.chunk.list = false\n" +
+          "3. ozone.client.stream.putblock.piggybacking = false\n" +
+          "4. ozone.client.key.write.concurrency = 1\n" +
+          "\n" +
+          "A warning message will be printed if any of the above configs are overridden by this.",
+      tags = ConfigTag.CLIENT)
+  private boolean hbaseEnhancementsAllowed = false;
+
+  // ozone.client.incremental.chunk.list
   @Config(key = "incremental.chunk.list",
-      defaultValue = "true",
+      defaultValue = "false",
       type = ConfigType.BOOLEAN,
       description = "Client PutBlock request can choose incremental chunk " +
           "list rather than full chunk list to optimize performance. " +
-          "Critical to HBase. EC does not support this feature.",
+          "Critical to HBase. EC does not support this feature. " +
+          "Can be enabled only when ozone.client.hbase.enhancements.allowed = true",
       tags = ConfigTag.CLIENT)
-  private boolean incrementalChunkList = true;
+  private boolean incrementalChunkList = false;
 
+  // ozone.client.stream.putblock.piggybacking
   @Config(key = "stream.putblock.piggybacking",
-          defaultValue = "true",
+          defaultValue = "false",
           type = ConfigType.BOOLEAN,
-          description = "Allow PutBlock to be piggybacked in WriteChunk " +
-                  "requests if the chunk is small.",
+          description = "Allow PutBlock to be piggybacked in WriteChunk requests if the chunk is small. " +
+              "Can be enabled only when ozone.client.hbase.enhancements.allowed = true",
           tags = ConfigTag.CLIENT)
-  private boolean enablePutblockPiggybacking = true;
+  private boolean enablePutblockPiggybacking = false;
 
+  // ozone.client.key.write.concurrency
   @Config(key = "key.write.concurrency",
       defaultValue = "1",
       description = "Maximum concurrent writes allowed on each key. " +
           "Defaults to 1 which matches the behavior before HDDS-9844. " +
-          "For unlimited write concurrency, set this to -1 or any negative integer value.",
+          "For unlimited write concurrency, set this to -1 or any negative integer value. " +
+          "Any value other than 1 is effective only when ozone.client.hbase.enhancements.allowed = true",
       tags = ConfigTag.CLIENT)
   private int maxConcurrentWritePerKey = 1;
 
@@ -298,6 +319,34 @@ public void validate() {
           OzoneConfigKeys.OZONE_CLIENT_BYTES_PER_CHECKSUM_MIN_SIZE;
     }
 
+    // Verify client configs related to HBase enhancements
+    // Enforce check on ozone.client.hbase.enhancements.allowed
+    if (!hbaseEnhancementsAllowed) {
+      // ozone.client.hbase.enhancements.allowed = false
+      if (incrementalChunkList) {
+        LOG.warn("Ignoring ozone.client.incremental.chunk.list = true " +
+            "because HBase enhancements are disallowed. " +
+            "To enable it, set ozone.client.hbase.enhancements.allowed = true.");
+        incrementalChunkList = false;
+        LOG.debug("Final ozone.client.incremental.chunk.list = {}", incrementalChunkList);
+      }
+      if (enablePutblockPiggybacking) {
+        LOG.warn("Ignoring ozone.client.stream.putblock.piggybacking = true " +
+            "because HBase enhancements are disallowed. " +
+            "To enable it, set ozone.client.hbase.enhancements.allowed = true.");
+        enablePutblockPiggybacking = false;
+        LOG.debug("Final ozone.client.stream.putblock.piggybacking = {}", enablePutblockPiggybacking);
+      }
+      if (maxConcurrentWritePerKey != 1) {
+        LOG.warn("Ignoring ozone.client.key.write.concurrency = {} " +
+            "because HBase enhancements are disallowed. " +
+            "To enable it, set ozone.client.hbase.enhancements.allowed = true.",
+            maxConcurrentWritePerKey);
+        maxConcurrentWritePerKey = 1;
+        LOG.debug("Final ozone.client.key.write.concurrency = {}", maxConcurrentWritePerKey);
+      }
+      // Note: ozone.fs.hsync.enabled is enforced by OzoneFSUtils#canEnableHsync, not here
+    }
   }
 
   public long getStreamBufferFlushSize() {
@@ -486,6 +535,14 @@ public void setDatastreamPipelineMode(boolean datastreamPipelineMode) {
     this.datastreamPipelineMode = datastreamPipelineMode;
   }
 
+  public void setHBaseEnhancementsAllowed(boolean isHBaseEnhancementsEnabled) {
+    this.hbaseEnhancementsAllowed = isHBaseEnhancementsEnabled;
+  }
+
+  public boolean getHBaseEnhancementsAllowed() {
+    return this.hbaseEnhancementsAllowed;
+  }
+
   public void setIncrementalChunkList(boolean enable) {
     this.incrementalChunkList = enable;
   }

diff --git a/...p-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/AbstractCommitWatcher.java b/...p-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/AbstractCommitWatcher.java
@@ -124,7 +124,6 @@ XceiverClientReply watchOnLastIndex() throws IOException {
    *
    * @param commitIndex log index to watch for
    * @return minimum commit index replicated to all nodes
-   * @throws IOException IOException in case watch gets timed out
    */
   CompletableFuture<XceiverClientReply> watchForCommitAsync(long commitIndex) {
     final MemoizedSupplier<CompletableFuture<XceiverClientReply>> supplier

diff --git a/...p-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java b/...p-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
@@ -364,7 +364,6 @@ public void writeOnRetry(long len) throws IOException {
    * it is a no op.
    * @param bufferFull flag indicating whether bufferFull condition is hit or
    *              its called as part flush/close
-   * @return minimum commit index replicated to all nodes
    * @throws IOException IOException in case watch gets timed out
    */
   public void watchForCommit(boolean bufferFull) throws IOException {