Add IThreadPoolWorkItem support to ThreadPoolDispatcher

[Aaronontheweb]

Significantly improves the message processing throughput of actors by reducing delegate invocations and taking advantage of thread locality.

Edit: and by thread locality, I mean significantly reduced context-switching for actors.

Changes

Added IThreadPoolWorkItem to all IRunnable implementations.

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

• This change follows the Akka.NET API Compatibility Guidelines.
• This change follows the Akka.NET Wire Compatibility Guidelines.
• I have reviewed my own pull request.
• Changes in public API reviewed, if any.

Latest dev Benchmarks

Using the in-memory PingPong benchmark on .NET 6

λ  dotnet run -c Release --framework net6.0                                                     
Warming up...                                                                                   
OSVersion:              Microsoft Windows NT 10.0.19044.0                                       
ProcessorCount:         16                                                                      
ClockSpeed:             0 MHZ                                                                   
Actor Count:            32                                                                      
Messages sent/received: 30000000  (3e7)                                                         
Is Server GC:           True                                                                    
Thread count:           34                                                                      
                                                                                                
ActorBase    first start time:  8.91 ms                                                         
ReceiveActor first start time: 45.73 ms                                                         
                                                                                                
            ActorBase                          ReceiveActor                                     
Throughput, Msgs/sec, Start [ms], Total [ms],  Msgs/sec, Start [ms], Total [ms]                 
         1,  8984000,     392.07,    3731.11,   9063000,      78.25,    3388.84                 
         5, 20891000,       5.94,    1442.00,  20297000,      16.24,    1494.82                 
        10, 25488000,       5.20,    1182.55,  28011000,     126.81,    1198.50                 
        15, 32017000,     140.92,    1078.92,  32362000,     109.97,    1037.81                 
        20, 37500000,     175.04,     975.45,  35714000,      13.92,     854.25                 
        30, 40816000,     255.00,     990.98,  39525000,      63.33,     823.27                 
        40, 46948000,      16.06,     655.99,  36231000,     253.94,    1082.63                 
        50, 40376000,     203.13,     946.64,  42553000,      29.44,     735.20                 
        60, 43352000,     205.82,     898.21,  38860000,     191.01,     963.05                 
        70, 42857000,     368.52,    1068.80,  44313000,     127.62,     805.22                 
        80, 46511000,      95.58,     741.49,  37974000,      93.35,     884.17                 
        90, 40760000,     159.32,     895.79,  39062000,      30.40,     798.53                 
       100, 54744000,     175.56,     724.27,  42075000,      31.63,     744.97                 
       200, 50847000,      33.61,     624.30,  41493000,     159.25,     882.35                 
       300, 42979000,     317.80,    1015.90,  45248000,      77.66,     741.33                 
       400, 53667000,     205.54,     764.83,  54347000,     332.45,     884.69                 
       500, 50000000,     110.50,     711.26,  50000000,     143.55,     744.52                 
       600, 52816000,       6.42,     574.99,  46948000,     141.28,     780.67                 
       700, 44182000,     159.14,     839.08,  43415000,     239.19,     930.38                 
       800, 53859000,     220.54,     777.80,  54347000,      19.47,     572.41                 
       900, 46012000,     175.21,     827.35,  54545000,       4.37,     554.97                 
Done..                                                                                          

This PR's Benchmarks

Warming up...                                                                            
OSVersion:              Microsoft Windows NT 10.0.19044.0                                
ProcessorCount:         16                                                               
ClockSpeed:             0 MHZ                                                            
Actor Count:            32                                                               
Messages sent/received: 30000000  (3e7)                                                  
Is Server GC:           True                                                             
Thread count:           34                                                               
                                                                                         
ActorBase    first start time:  8.28 ms                                                  
ReceiveActor first start time: 39.51 ms                                                  
                                                                                         
            ActorBase                          ReceiveActor                              
Throughput, Msgs/sec, Start [ms], Total [ms],  Msgs/sec, Start [ms], Total [ms]          
         1, 25231000,     391.25,    1581.24,  27051000,     440.64,    1549.65          
         5, 44444000,     158.10,     833.48,  41265000,     347.83,    1075.76          
        10, 57361000,     222.17,     745.25,  44052000,     187.32,     869.20          
        15, 61099000,     207.29,     698.31,  43795000,     190.95,     876.44          
        20, 63291000,      31.30,     505.77,  50083000,     206.13,     805.89          
        30, 65359000,     284.12,     743.75,  56390000,     221.03,     753.66          
        40, 65934000,      15.41,     471.25,  61728000,      94.93,     580.98          
        50, 67567000,      31.59,     475.65,  63025000,     236.08,     713.05          
        60, 60728000,     238.08,     732.83,  62111000,     299.63,     783.29          
        70, 67873000,       4.40,     446.66,  50335000,      95.15,     692.11          
        80, 60362000,     333.11,     830.40,  53003000,     111.07,     677.60          
        90, 62240000,     222.33,     705.28,  55762000,     141.14,     679.98          
       100, 68181000,       4.37,     444.53,  60120000,     285.07,     784.31          
       200, 68649000,     142.25,     579.99,  58365000,      79.38,     593.49          
       300, 70422000,     204.35,     631.35,  63559000,     111.01,     583.45          
       400, 61728000,     285.17,     771.91,  62370000,     126.93,     608.64          
       500, 63965000,       8.86,     478.63,  46082000,     222.23,     873.65          
       600, 50675000,      63.59,     655.66,  64239000,     267.16,     734.81          
       700, 67264000,     127.18,     573.78,  59171000,     143.44,     650.65          
       800, 52539000,     126.29,     697.53,  59405000,     283.21,     788.38          
       900, 69605000,     175.30,     606.64,  43795000,     174.37,     859.86          

[Aaronontheweb]

Major change here - leverages new overload to the ThreadPool.UnsafeQueueUserWorkItem method added in .NET Core 3.0: https://docs.microsoft.com/en-us/dotnet/api/system.threading.threadpool.unsafequeueuserworkitem?view=net-6.0

[Aaronontheweb]

So Akka.Remote can't take advantage of this without:

1. Changing the channel-executor to use the ThreadPool.UnsafeQueueUserWorkItem internally or
2. Running on the default dispatcher, which is less than ideal for various reasons.

That being said, I'm going to benchmark it.

[Arkatufus]

LGTM