Failure to form a cluster on .NET 9 when using Azure discovery with Akka.Management 1.5.35

[object]

Version Information
Akka.Management
Akka.Discovery.Azure
Akka.Coordination.Azure
Version 1.5.35
.NET 9

Describe the bug
Services start but unable to form a cluster continuously raising the following exception:

  "Message": "Failed to create/retrieve self discovery entry, retrying.",
  "MessageTemplate": "{Message:l}",
  "Renderings": [
    "Failed to create/retrieve self discovery entry, retrying."
  ],
  "Exception": {
    "Type": "FileNotFoundException",
    "Message": "Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.",
    "StackTrace": "   at Akka.Discovery.Azure.ClusterMemberTableClient.GetEntityAsync(String rowKey, CancellationToken token)\r\n   at Akka.Discovery.Azure.ClusterMemberTableClient.GetOrCreateAsync(String host, IPAddress address, Int32 port, CancellationToken token)\r\n   at Akka.Discovery.Azure.Actors.AzureDiscoveryGuardian.<Initializing>b__24_1(CancellationToken token)\r\n   at Akka.Discovery.Azure.Actors.AzureDiscoveryGuardian.ExecuteOperationWithRetry[T](Func`2 operation)\r\n   at Akka.Actor.PipeToSupport.PipeTo[T](Task`1 taskToPipe, ICanTell recipient, Boolean useConfigureAwait, IActorRef sender, Func`2 success, Func`2 failure)"
  },
  "_Message": "Failed to create/retrieve self discovery entry, retrying.",
  "Thread": "0038",
  "LogSource": "AzureDiscoveryGuardian (akka://Oddjob)",
  "Timestamp": "2025-01-16T06:47:49.7561942Z",
  "ActorPath": "akka://Oddjob/system/azure-discovery-guardian",
  "SourceContext": "Akka.Event.DummyClassForStringSources",

To Reproduce
Build applications using Akka.Management 1.5.35, run on .NET 9.

Environment
Linux with .NET SDK 9.0

Additional context
After downgrading Akka.Management packages to the version 1.5.33 the problem goes away.

[Aaronontheweb]

We shouldn't even be referencing this package with the distributions we emit for .NET 6+ - this is a .NET Standard thing

[Aaronontheweb]

This should be addressed via akkadotnet/akka.net#7482

[Aaronontheweb]

We are now in the process of updating every library to use Akka.NET v1.5.37, which rolls back to the BCL versions to 6.0.

[object]

I see on Nuget only main Akka packages updated, not Sql or Azure. Will you also update them?

[object]

I have rebuilt our services using whatever Akka package I could find with version 1.5.37. Now they can't form a cluster ("No coordinator found to register. "Probably, no seed-nodes configured and manual cluster join not performed?")

And this:

Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

  | Exception_StackTrace | at Akka.Discovery.Azure.ClusterMemberTableClient.GetEntityAsync(String rowKey, CancellationToken token) at Akka.Discovery.Azure.ClusterMemberTableClient.GetOrCreateAsync(String host, IPAddress address, Int32 port, CancellationToken token) at Akka.Discovery.Azure.Actors.AzureDiscoveryGuardian.b__24_1(CancellationToken token) at Akka.Discovery.Azure.Actors.AzureDiscoveryGuardian.ExecuteOperationWithRetry[T](Func2 operation) at Akka.Actor.PipeToSupport.PipeTo[T](Task1 taskToPipe, ICanTell recipient, Boolean useConfigureAwait, IActorRef sender, Func2 success, Func2 failure)

Was I too quick and should have waited for more packages to come?

[Aaronontheweb]

Yes, I'd hold your horses and wait for the entire package train to get updated. We're making sure everything is dual targeted and are fixing areas where we manually upgraded the BCL in some places. This is going to take all day.

[Aaronontheweb]

Fixing the dual targeting stuff also exposed some problems / API gaps in libraries like Akka.Persistence.Sql - this is a consequence of not using a giant MonoRepo for everything (and that invites its own set of problems)

[object]

Yes, with multiple repos things become more fragile. I will wait until everything is updated to 1.5.37. But Phobos doesn't need a new release, right?

[Aaronontheweb]

But Phobos doesn't need a new release, right?

Probably not but we're going to do one any way out of an abundance of caution

Yes, with multiple repos things become more fragile.

Overall it's supposed to reduce the testing burden for each plugin by "pricing in" the QA on the original libraries as an output artifact, but stuff like having inconsistent package targeting across dependencies is a problem.

[Aaronontheweb]

although in this case we screwed up by having the .NET Standard version use a higher "version" of the BCL than what we use on .NET 6 - that was the real mistake. We could have either fixed it by dropping .NET 6 and moving to .NET 8 or by rolling back the .NET Standard 6->8 upgrade. We're choosing the latter.

[object]

I wonder why you didn't choose the first option - just drop .NET 6 and move to .NET 8.