[Issue 9195] Evict silos from cluster if they remain in the Joining or Created state for longer than MaxJoinAttemptTime

[Chris-Eckhardt]

Fixes #9195

In this PR:

• Adds the flag ClusterMembershipOptions.EvictWhenMaxJoinAttemptTimeExceeded (default to true)
• Checks MembershipTableSnapshot for silos in a state of Created or Joined for longer than ClusterMembershipOptions.MaxJoinAttemptTime

Microsoft Reviewers: Open in CodeFlow

[Chris-Eckhardt]

@dotnet-policy-service agree

[Chris-Eckhardt]

Currently looking at what tests I can modify to cover this or If I need to add a new one.

[ReubenBond]

Currently looking at what tests I can modify to cover this or If I need to add a new one.

ClusterHealthMonitorTests.cs may be a good fit. The test case there is large (too large), but adding a new test case or modifying the existing one is ok

[Chris-Eckhardt]

I added a test with numVotesForDeathDeclaration == 1. If I set it to anything higher the test will fail, I need to address that.

[Chris-Eckhardt]

I ended up refactoring the tests in ClusterHealthMonitorTests.cs to be able to cover multiple scenarios.

Tests added:

• EvictWhenMaxJoinAttemptTimeExceeded is disabled
• NumVotesForDeathDeclaration == 1
• NumVotesForDeathDeclaration == 2
• NumVotesForDeathDeclaration == 3

Changes Related to the refactor:

• Created private ClusterHealthMonitorTestRig class
• Both ClusterHealthMonitor_BasicScenario_Runner and the new ClusterHealthMonitor_StaleJoinOrCreatedSilos_Runner use this internally now.

        private class ClusterHealthMonitorTestRig(
            MembershipTableManager manager,
            IClusterMembershipService membershipService,
            IOptionsMonitor<ClusterMembershipOptions> optionsMonitor,
            ClusterHealthMonitor.ITestAccessor testAccessor)
        {
            public readonly MembershipTableManager Manager = manager;
            public readonly IClusterMembershipService MembershipService = membershipService;
            public readonly IOptionsMonitor<ClusterMembershipOptions> OptionsMonitor = optionsMonitor;
            public readonly ClusterHealthMonitor.ITestAccessor TestAccessor = testAccessor;
        }

[benjaminpetit]

Thanks, that would help a lot when recovering a failed silo!