LFU after access fails to update expiry when read buffer is not full

[bitfaster]

From the discussion here #516, this is the repro:

[Fact]
public void Repro()
{ 
    var cache = new ConcurrentLfuBuilder<int, int>()
        .WithCapacity(10)
        .WithExpireAfterAccess(TimeSpan.FromSeconds(5))
        .Build();

    cache.AddOrUpdate(1, 1);

    Thread.Sleep(4000);

    cache.TryGet(1, out var value).Should().BeTrue(); // maintenance doesn't run here

    Thread.Sleep(2000);

    cache.TryGet(1, out value).Should().BeTrue();
    cache.TryGet(1, out value).Should().BeTrue();
}

Since maintenance only runs when the read buffer is full, expiry time is never updated.

Solution:

• Policy computes new expiry time at the call site (either on read or update), instead of during maintenance.
• For read, IsExpired has just been called, so we can piggyback on the current time to avoid two calls
• We can no longer avoid re-scheduling the same node as part of maintenance (previously we checked if expiry time had changed). Likely the removal of calling expiry calculator is equal cost.

There are a couple of further optimizations:

1. For expire after access, we could make a policy which has empty OnRead and AfterRead methods, enabling the JIT to elide the method calls to the policy and the expiry calculator, which are anyway a no-op.
2. During maintenance, it may be worth testing the node we are now scheduling against the last node we scheduled. For cases where the same node is read many times, this may avoid duplicate scheduling. Need to measure whether adding this is actually beneficial.

[coveralls]

coverage: 99.209% (+0.07%) from 99.141%
when pulling ac459e7 on users/alexpeck/lfuafteraccess
into 07f8a1c on main.

[bitfaster]

For 4ee9397 the JIT does not completely elide the additional branch, there is an extra and (.NET 6 shown)

.NET Fwk has 4 extra instructions:

[bitfaster]

.NET 6 generates identical assembly:

.NET Fwk, 4 bytes extra code within TryGetImpl. It looks like this is really only 1 extra instruction because the disassembly has a bunch of dupes (bug in benchmark dot net):