Bootstrap FC optimization: Interfaces for state build and state transition flows #2946

piyujai · 2024-11-20T09:18:14Z

No description provided.

piyujai · 2024-11-20T09:38:37Z

ambry-api/src/main/java/com/github/ambry/config/ClusterMapConfig.java

@@ -372,6 +372,11 @@ public class ClusterMapConfig {
  @Default("false")
  public final boolean clusterMapIgnoreDownwardStateTransition;

+  public static final String ENABLE_FILE_COPY_FOR_BOOTSTRAP = "clustermap.enable.file.copy.for.bootstrap";
+  @Config(ENABLE_FILE_COPY_FOR_BOOTSTRAP)
+  @Default("false")


Merge into single config

piyujai · 2024-11-21T05:28:06Z

ambry-store/src/main/java/com/github/ambry/store/StorageManager.java

+        throw new StateTransitionException(
+            "Existing Disk manager not found for replica " + partitionName + " is not found in clustermap for " + currentNode, DiskManagerNotFoundForFileCopyStateBuild);
+      }
+      diskManager.addBlobStoreToCompactionManager((BlobStore) store);


store.start() and diskManager.addBlobStoreToCompactionManager((BlobStore) store) can be moved to file copy manager class later: Justin's suggestion

Can you elaborate as to why would that be advantageous?

DevenAhluwalia · 2024-11-24T14:35:44Z

ambry-api/src/main/java/com/github/ambry/config/ClusterMapConfig.java

+  public static final String ENABLE_FILE_COPY_FOR_BOOTSTRAP = "clustermap.enable.file.copy.for.bootstrap";
+  @Config(ENABLE_FILE_COPY_FOR_BOOTSTRAP)
+  @Default("false")
+  public final boolean enableFileCopyForBootstrap;


Keeping the nomenclature in this file consistent, the name of this key would be
clusterMapEnableFileCopyForBootstrap

DevenAhluwalia · 2024-11-24T14:37:40Z

ambry-api/src/main/java/com/github/ambry/config/StoreConfig.java

+  @Config(ENABLE_FILE_COPY_FOR_BOOTSTRAP)
+  @Default("false")
+  public final boolean enableFileCopyForBootstrap;
+  public static final String ENABLE_FILE_COPY_FOR_BOOTSTRAP = "clustermap.enable.file.copy.for.bootstrap";


Either this or the the key defined in ClusterMapConfig needs to be removed.
Further,

Key in this file will have store as prefix

Keeping the nomenclature in this file consistent, the key name would be storeEnableFileCopyForBootstrap

DevenAhluwalia · 2024-11-24T14:43:12Z

ambry-store/src/main/java/com/github/ambry/store/DiskManager.java

+        if (!storeConfig.enableFileCopyForBootstrap) {
+          store.start();
+        }
+
        // collect store segment requirements and add into DiskSpaceAllocator
        List<DiskSpaceRequirements> storeRequirements = Collections.singletonList(store.getDiskSpaceRequirements());


If store.start() hasn't been invoked, in the normal Replication protocol, would store.getDiskSpaceRequirements() work as expected?

DevenAhluwalia · 2024-11-24T14:45:09Z

ambry-store/src/main/java/com/github/ambry/store/StorageManager.java

@@ -755,6 +760,8 @@ public void onPartitionBecomeBootstrapFromOffline(String partitionName) {
          }
        } while (!replicaAdded);

+


nit: remove unnecessary new lines.

Applicable throughout the diff wherever applicable.

DevenAhluwalia · 2024-11-24T14:50:48Z

ambry-store/src/main/java/com/github/ambry/store/StorageManager.java

+        throw new StateTransitionException(
+            "Existing Disk manager not found for replica " + partitionName + " is not found in clustermap for " + currentNode, DiskManagerNotFoundForFileCopyStateBuild);
+      }
+      diskManager.addBlobStoreToCompactionManager((BlobStore) store);


Can you elaborate as to why would that be advantageous?

DevenAhluwalia · 2024-11-24T14:53:24Z

This PR needs to be opened with target branch bootstrap-optimisation-lld-flow

piyujai · 2024-11-28T10:24:22Z

ambry-api/src/main/java/com/github/ambry/config/StoreConfig.java

@@ -674,6 +674,13 @@ public class StoreConfig {
  public final boolean storeBlockStaleBlobStoreToStart;
  public final static String storeBlockStaleBlobStoreToStartName = "store.block.stale.blob.store.to.start";

+
+  @Config(STORE_ENABLE_FILE_COPY_FOR_BOOTSTRAP)


use serverReplicationProtocolForHydration

piyujai · 2024-11-28T10:34:22Z

ambry-file-transfer/src/main/java/com/github/ambry/FileCopyManager.java

@@ -41,7 +41,8 @@ class PartitionStateChangeListenerImpl implements PartitionStateChangeListener {

    @Override
    public void onPartitionBecomeBootstrapFromOffline(String partitionName) {
-
+      // StateBuilding at the end of FCM's Offline->Bootstrap transition
+      storageManagerListener.buildStateForFileCopy(partitionName);


move to storeManager

…ition flows (#2946) * Changes for bootstrap improvement with statebuild * Added initial changes for state build * Changes for moving state build to FCM * Changes for state build including FCM invocation * reformatting changes --------- Co-authored-by: Jai Balani <jbalani@jbalani-mn1.linkedin.biz>

piyujai changed the title ~~Base changes for Offline-~~ Base changes for Offline->Bootstrap Nov 20, 2024

piyujai changed the title ~~Base changes for Offline->Bootstrap~~ Bootstrap FC optimization: Base changes for Offline->Bootstrap Nov 20, 2024

piyujai changed the title ~~Bootstrap FC optimization: Base changes for Offline->Bootstrap~~ [WIP] Bootstrap FC optimization: Base changes for Offline->Bootstrap Nov 20, 2024

piyujai force-pushed the bootstrap_state_build_changes branch from b251c8c to 8b13123 Compare November 20, 2024 09:22

piyujai commented Nov 20, 2024

View reviewed changes

piyujai force-pushed the bootstrap_state_build_changes branch 2 times, most recently from d857928 to b4860e8 Compare November 20, 2024 10:27

piyujai marked this pull request as draft November 20, 2024 19:45

piyujai commented Nov 21, 2024

View reviewed changes

DevenAhluwalia requested changes Nov 24, 2024

View reviewed changes

piyujai force-pushed the bootstrap_state_build_changes branch from 6d54c0e to 08af79a Compare November 26, 2024 09:08

Changes for bootstrap improvement with statebuild

24b819c

piyujai force-pushed the bootstrap_state_build_changes branch from 08af79a to d3ef815 Compare November 26, 2024 09:12

piyujai changed the base branch from master to bootstrap-optimisation-lld-flow November 26, 2024 09:13

piyujai force-pushed the bootstrap_state_build_changes branch from d3ef815 to b2385bc Compare November 26, 2024 09:17

piyujai changed the title ~~[WIP] Bootstrap FC optimization: Base changes for Offline->Bootstrap~~ [WIP] Bootstrap FC optimization: Interfaces for state build Nov 26, 2024

piyujai changed the title ~~[WIP] Bootstrap FC optimization: Interfaces for state build~~ [WIP] Bootstrap FC optimization: Interfaces for state build and state transition flows Nov 26, 2024

piyujai force-pushed the bootstrap_state_build_changes branch from b2385bc to 720c445 Compare November 26, 2024 09:44

Added initial changes for state build

e9b1d5a

piyujai force-pushed the bootstrap_state_build_changes branch from 720c445 to e9b1d5a Compare November 26, 2024 09:44

piyujai marked this pull request as ready for review November 26, 2024 09:45

piyujai force-pushed the bootstrap_state_build_changes branch from f9b9cc8 to 1d17ad8 Compare November 28, 2024 09:01

piyujai changed the title ~~[WIP] Bootstrap FC optimization: Interfaces for state build and state transition flows~~ Bootstrap FC optimization: Interfaces for state build and state transition flows Nov 28, 2024

piyujai commented Nov 28, 2024

View reviewed changes

piyujai force-pushed the bootstrap_state_build_changes branch from 1d17ad8 to 80d1bce Compare November 28, 2024 10:38

Changes for moving state build to FCM

1124d46

piyujai force-pushed the bootstrap_state_build_changes branch from 80d1bce to 1124d46 Compare November 28, 2024 11:08

Changes for state build including FCM invocation

7942181

reformatting changes

6e10bcf

piyujai force-pushed the bootstrap_state_build_changes branch from a621b02 to 6e10bcf Compare November 28, 2024 14:16

DevenAhluwalia approved these changes Nov 28, 2024

View reviewed changes

piyujai merged commit 944356e into bootstrap-optimisation-lld-flow Nov 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootstrap FC optimization: Interfaces for state build and state transition flows #2946

Bootstrap FC optimization: Interfaces for state build and state transition flows #2946

piyujai commented Nov 20, 2024

piyujai Nov 20, 2024 •

edited

Loading

piyujai Nov 21, 2024 •

edited

Loading

DevenAhluwalia Nov 24, 2024

DevenAhluwalia Nov 24, 2024

DevenAhluwalia Nov 24, 2024

piyujai Nov 26, 2024

DevenAhluwalia Nov 24, 2024

DevenAhluwalia Nov 24, 2024

DevenAhluwalia Nov 24, 2024

DevenAhluwalia Nov 24, 2024

DevenAhluwalia commented Nov 24, 2024

piyujai Nov 28, 2024

piyujai Nov 28, 2024

		@@ -755,6 +760,8 @@ public void onPartitionBecomeBootstrapFromOffline(String partitionName) {
		}
		} while (!replicaAdded);

Bootstrap FC optimization: Interfaces for state build and state transition flows #2946

Bootstrap FC optimization: Interfaces for state build and state transition flows #2946

Conversation

piyujai commented Nov 20, 2024

piyujai Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

piyujai Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DevenAhluwalia commented Nov 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piyujai Nov 20, 2024 •

edited

Loading

piyujai Nov 21, 2024 •

edited

Loading