Akka.Cluster: enable `keep-majority` default SBR #6628

Aaronontheweb · 2023-03-30T16:44:32Z

Changes

Enables keep-majority as the default SBR and turns it on by default for Akka.Cluster. This was a planned change for Akka.NET v1.5.0 but we didn't implement it.

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

This change follows the Akka.NET API Compatibility Guidelines.
This change follows the Akka.NET Wire Compatibility Guidelines.
I have reviewed my own pull request.
I have added documentation for this PR.

Enables `keep-majority` as the default SBR and turns it on by default for Akka.Cluster. This was a planned change for Akka.NET v1.5.0 but we didn't implement it.

Aaronontheweb · 2023-03-30T16:44:51Z

I think I need to write some updates to the documentation and a release advisory for this change as well

Aaronontheweb · 2023-03-30T16:56:01Z

Looks like I need to update the DowningProviderSpec as well

Aaronontheweb

Detailed my changes

Aaronontheweb · 2023-04-04T19:57:18Z

src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCluster.DotNet.verified.txt

@@ -18,6 +18,7 @@
 [assembly: System.Runtime.Versioning.TargetFrameworkAttribute(".NETCoreApp,Version=v6.0", FrameworkDisplayName=".NET 6.0")]
 namespace Akka.Cluster
 {
+    [System.ObsoleteAttribute("No longer used as of Akka.NET v1.5.2")]


AutoDowning is now obsolete - it was a bad idea anyway.

Aaronontheweb · 2023-04-04T19:57:42Z

src/contrib/cluster/Akka.Cluster.Tools.Tests/Singleton/ClusterSingletonConfigSpec.cs

@@ -37,7 +37,7 @@ public void ClusterSingletonManagerSettings_must_have_default_config()
            clusterSingletonManagerSettings.SingletonName.ShouldBe("singleton");
            clusterSingletonManagerSettings.Role.ShouldBe(null);
            clusterSingletonManagerSettings.HandOverRetryInterval.TotalSeconds.ShouldBe(1);
-            clusterSingletonManagerSettings.RemovalMargin.TotalSeconds.ShouldBe(0);
+            clusterSingletonManagerSettings.RemovalMargin.TotalSeconds.ShouldBe(20); // now 20 due to default SBR settings


RemovalMargin now uses SBR default value of 20 seconds (this is where the value is computed from.)

Aaronontheweb · 2023-04-04T19:58:00Z

docs/community/whats-new/akkadotnet-v1.5-upgrade-advisories.md

@@ -11,6 +11,48 @@ This document contains specific upgrade suggestions, warnings, and notices that
 <iframe width="560" height="315" src="https://www.youtube.com/embed/-UPestlIw4k" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
 <!-- markdownlint-enable MD033 -->

+## Upgrading to Akka.NET v1.5.2


Upgrade advisories for this PR as well as #6389

Aaronontheweb · 2023-04-04T19:58:32Z

docs/community/whats-new/akkadotnet-v1.5-upgrade-advisories.md

+
+#### Disabling the Default Downing Provider
+
+To disable the default Akka.Cluster downing provider, simply configure the following in your HOCON:


Duplicated this section from the Split Brain Resolver page - explains how to disable the new default SBR, since I'm fairly certain we'll get a question about that in the future.

Aaronontheweb · 2023-04-04T19:58:53Z

docs/articles/clustering/split-brain-resolver.md

@@ -38,6 +38,19 @@ Keep in mind that split brain resolver will NOT work when `akka.cluster.auto-dow

 Beginning in Akka.NET v1.4.16, the Akka.NET project has ported the original split brain resolver implementations from Lightbend as they are now open source. The following section of documentation describes how Akka.NET's hand-rolled split brain resolvers are implemented.

+> [!IMPORTANT]
+> As of Akka.NET v1.5.2, the `keep-majority` split brain resolution strategy is now enabled by default. This should be acceptable for the majority of Akka.Cluster users, but please read on.


Tried to make it clear that there is now a default SBR enabled.

src/core/Akka.Cluster/ClusterDaemon.cs

Aaronontheweb · 2023-04-04T20:02:47Z

src/core/Akka.Cluster/ClusterSettings.cs

-                DowningProviderType = typeof(AutoDowning);
-            else
-                DowningProviderType = typeof(NoDowning);
+            DowningProviderType = !string.IsNullOrEmpty(downingProviderClassName) ? Type.GetType(downingProviderClassName, true) : typeof(NoDowning);


Disabled loading of AutoDowning here as it's no longer supported.

Aaronontheweb · 2023-04-04T20:03:02Z

src/core/Akka.Cluster/Configuration/Cluster.conf

-    # formed in case of network partition.
-    # Disable with "off" or specify a duration to enable auto-down.
-	# If a downing-provider-class is configured this setting is ignored.
-    auto-down-unreachable-after = off


Removed auto-down-unreachable-after setting from the base config.

Aaronontheweb · 2023-04-04T20:03:15Z

src/core/Akka.Cluster/Configuration/Cluster.conf

@@ -63,7 +55,9 @@ akka {
    # `Akka.Cluster.IDowningProvider` having two argument constructor:
    #   - argument 1: accepting an `ActorSystem`
    #   - argument 2: accepting an `Akka.Cluster.Cluster`
-    downing-provider-class = ""
+    downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"


This is the major change - the default SBR.

Aaronontheweb · 2023-04-04T20:03:43Z

src/core/Akka.Remote.TestKit/MultiNodeSpec.cs

@@ -379,6 +379,7 @@ public static Config BaseConfig
                            }
                          }
                        }
+                        cluster.downing-provider-class = """" #disable SBR by default


Backwards compat for the MNTR - keep the SBR disabled by default. Most tests probably don't want the SBR running anyway unless they explicitly enable it.

Aaronontheweb · 2023-04-05T14:47:56Z

src/core/Akka.Cluster/ClusterSettings.cs

            AutoDownUnreachableAfter = clusterConfig.GetTimeSpanWithOffSwitch("auto-down-unreachable-after");
+            if (AutoDownUnreachableAfter == default(TimeSpan)) // need to restore correct defaults now that it's been deleted from HOCON file


This is a work-around needed to restore the "correct" defaults for AutoDownUnreachableAfter now that it's no longer included in the default cluster.conf file.

Aaronontheweb · 2023-04-05T15:13:09Z

src/contrib/cluster/Akka.Cluster.Sharding.Tests.MultiNode/ClusterShardingSpec.cs

@@ -58,7 +58,9 @@ public class ClusterShardingSpecConfig : MultiNodeClusterShardingConfig
            CommonConfig = ConfigurationFactory.ParseString($@"
                akka.cluster.sharding.verbose-debug-logging = on
                #akka.loggers = [""akka.testkit.SilenceAllTestEventListener""]
-
+                akka.cluster.downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""


This spec previously relied on auto-down-unreachable-after - in order to make it pass again we needed to:

Enable the SBR

Set the akka.cluster.split-brain-resolver.stable-after to the lowest possible value (must be > 0.0s)

Set the akka.cluster.down-removal-margin to the lowest possible value (must be > 0.0s)

Arkatufus

Just have a question

Arkatufus · 2023-04-05T15:42:31Z

docs/articles/clustering/split-brain-resolver.md

+To disable the default Akka.Cluster downing provider, simply configure the following in your HOCON:
+
+```hocon
+akka.cluster.downing-provider-class = ""


Question, since we removed AutoDowning, what will happen if a user turn SBR off?

Answered in chat - nothing. AutoDowning was never enabled by default and we strongly discouraged users from ever enabling it. If the default SBR is disabled Akka.Cluster behaves exactly like it does today: unreachable nodes stay unreachable until downed manually by pbm or some other process.

Aaronontheweb · 2023-04-05T17:00:55Z

Local test runner appears to be crashing on Windows - I'll re-run it once the rest of the PR completes testing.

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

 ---> System.Exception: An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

Do not press `Update Branch`

Can't really run the Akka.Cluster.Sharding MNTR suite without it

… into default-SBR

enable keep-majority default SBR

9215e9b

Enables `keep-majority` as the default SBR and turns it on by default for Akka.Cluster. This was a planned change for Akka.NET v1.5.0 but we didn't implement it.

Aaronontheweb added akka-cluster configuration labels Mar 30, 2023

Aaronontheweb added this to the 1.5.2 milestone Mar 30, 2023

Merge branch 'dev' into default-SBR

fb42fda

Aaronontheweb added 5 commits April 4, 2023 14:30

added upgrade advisories to documentation and some spec / warning fixes

e9b08c2

fixed typos

3782132

added documentation on how to disable the default downing provider

7ca8104

added API approvals

6e74d88

disable SBR in MNTR

97eb1c2

Aaronontheweb changed the title ~~enable keep-majority default SBR~~ Akka.Cluster: enable keep-majority default SBR Apr 4, 2023

Merge branch 'dev' into default-SBR

24d46b9

Aaronontheweb marked this pull request as ready for review April 4, 2023 19:56

Update MultiNodeClusterSpec.cs

248404f

Aaronontheweb commented Apr 4, 2023

View reviewed changes

Aaronontheweb requested a review from Arkatufus April 4, 2023 20:04

Aaronontheweb added 3 commits April 4, 2023 19:20

fixed equality members on InitJoin

9df4d23

fix default auto-down-unreachable-after parse value

ed9918c

disable SBR in all clustering specs

b6a4ab5

Aaronontheweb commented Apr 5, 2023

View reviewed changes

Aaronontheweb added 3 commits April 5, 2023 09:55

cleanup

3df700b

reconfigured SBR for Akka.Cluster.Sharding specs

2563099

fixed - had to adjust down-removal-margin

5ab3acc

Aaronontheweb commented Apr 5, 2023

View reviewed changes

Merge branch 'dev' into default-SBR

658ab28

Arkatufus requested changes Apr 5, 2023

View reviewed changes

fixed SBR issues with Akka.Cluster.Sharding MNTR

6ee93e5

Arkatufus approved these changes Apr 5, 2023

View reviewed changes

Merge branch 'dev' into default-SBR

5337e46

Arkatufus enabled auto-merge (squash) April 5, 2023 16:49

Aaronontheweb added 4 commits April 5, 2023 12:51

restored auto-down-unreachable-after

251fade

Can't really run the Akka.Cluster.Sharding MNTR suite without it

Merge branch 'dev' into default-SBR

1705fc7

approave API changes

7eedcdd

Merge branch 'default-SBR' of https://github.com/Aaronontheweb/akka.net…

e45d60f

… into default-SBR

Aaronontheweb disabled auto-merge April 5, 2023 19:46

Aaronontheweb merged commit e53c7b0 into akkadotnet:dev Apr 5, 2023

Aaronontheweb deleted the default-SBR branch April 5, 2023 19:54

Arkatufus mentioned this pull request Apr 5, 2023

Update RELEASE_NOTES.md for v1.5.2 release #6635

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Akka.Cluster: enable `keep-majority` default SBR #6628

Akka.Cluster: enable `keep-majority` default SBR #6628

Aaronontheweb commented Mar 30, 2023 •

edited

Loading

Aaronontheweb commented Mar 30, 2023

Aaronontheweb commented Mar 30, 2023

Aaronontheweb left a comment

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 4, 2023

Aaronontheweb Apr 5, 2023

Aaronontheweb Apr 5, 2023

Arkatufus left a comment

Arkatufus Apr 5, 2023

Aaronontheweb Apr 5, 2023

Aaronontheweb commented Apr 5, 2023


		#### Disabling the Default Downing Provider

		To disable the default Akka.Cluster downing provider, simply configure the following in your HOCON:

		AutoDownUnreachableAfter = clusterConfig.GetTimeSpanWithOffSwitch("auto-down-unreachable-after");
		if (AutoDownUnreachableAfter == default(TimeSpan)) // need to restore correct defaults now that it's been deleted from HOCON file

Akka.Cluster: enable keep-majority default SBR #6628

Akka.Cluster: enable keep-majority default SBR #6628

Conversation

Aaronontheweb commented Mar 30, 2023 • edited Loading

Changes

Checklist

Aaronontheweb commented Mar 30, 2023

Aaronontheweb commented Mar 30, 2023

Aaronontheweb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arkatufus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aaronontheweb commented Apr 5, 2023

Do not press Update Branch

Akka.Cluster: enable `keep-majority` default SBR #6628

Akka.Cluster: enable `keep-majority` default SBR #6628

Aaronontheweb commented Mar 30, 2023 •

edited

Loading

Do not press `Update Branch`