HashedWheelTimer startup crash on .NET 6+

[Aaronontheweb]

Changes

More issues with the PeriodicTimer - this time it's a bit of a spooky heisenbug. Added a spec that can reproduce it but it must be run continuously in order to catch it.

Originally reproduced at petabridge/TurboMqtt#55

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

• This change follows the Akka.NET API Compatibility Guidelines.
• This change follows the Akka.NET Wire Compatibility Guidelines.
• I have reviewed my own pull request.

[Aaronontheweb]

I believe our problems were related to the combination of Interlocked and volatile both trying to access the same field across multiple cores - but I don't think I can prove it other than to say that since I made this change to the HashedWheelTimerScheduler I've run this test continuously for over 30 minutes (it runs about once a second) without any errors. Previously, it would fail in about 5 minutes.

I'll keep the thing running for a while longer and report back if the problem isn't fixed, but I think this should resolve it.

[Aaronontheweb]

Took a while but was able to make this crash even without the volatile - I'll go ahead and add that back then. I think what I'll do instead is just remove this else statement and have it loop back to the top via a goto.

edit: came up with a safer, less complicated solution

[Aaronontheweb]

Fixed this change by doing only one volatile read at the start of the if statement - been running my test case at a rate of once per second for about 2 hours and haven't seen the problem occur since.

[Aaronontheweb]

Enabled nullable for just this test - on my quest to ensure that we start doing that generally everywhere.

[Aaronontheweb]

This is the key fix - cache the value once so we're not doing a volatile read on each branch of the if..else - that's what's lead to the problems.

[Arkatufus]

LGTM

[Aaronontheweb]

The solution is good for merge now

[Arkatufus]

LGTM

[Arkatufus]

LGTM

[to11mtm]

Can't check the box from mobile but looks good

…

On Fri, Apr 26, 2024, 5:50 PM Gregorius Soedharmo ***@***.***> wrote: ***@***.**** approved this pull request. LGTM ------------------------------ In src/core/Akka/Actor/Scheduler/HashedWheelTimerScheduler.cs <#7174 (comment)>: > @@ -147,25 +147,27 @@ private static int NormalizeTicksPerWheel(int ticksPerWheel) private void Start() { - if (_workerState == WORKER_STATE_STARTED) + // only read the worker state once so it can't be a moving target for else-branch + var workerStateRead = _workerState; LGTM — Reply to this email directly, view it on GitHub <#7174 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC7UYVP5YCOMXRQ6AJLPWP3Y7LD3VAVCNFSM6AAAAABG3ICIBKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAMRVHEZDAMZUG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Aaronontheweb]

I believe our problems were related to the combination of Interlocked and volatile both trying to access the same field across multiple cores

this was wrong, btw - looks like it was the concurrent read / write access to volatile causing us to jump into the else statement that did it. Had nothing to do with Interlocked.

[Aaronontheweb]

 Failed Akka.Cluster.Metrics.Tests.MetricValuesSpec.NodeMetrics_MetricValues_should_extract_expected_metrics_for_load_balancing [7 s]
  Error Message:
   System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Akka.Cluster.Metrics.Tests.MetricValuesSpec.NodeMetrics_MetricValues_should_extract_expected_metrics_for_load_balancing() in D:\a\1\s\src\contrib\cluster\Akka.Cluster.Metrics.Tests\MetricSpec.cs:line 324
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
  Passed Akka.Cluster.Metrics.Tests.EWMASpec.EWMA_should_calculate_correct_ewma_for_normal_d

Not related to my changes - I have an open PR for investigating this one at home.