[C#] FasterLog - tailing iterator from different process

[AlgorithmsAreCool]

Can FasterLog be used by multiple processes in any configuration?

My primary use case would be a single Writer Process and a single Reader Process.

I'm ok with the reader being truly read-only and not being able to truncate.

[badrishc]

Sure, that should be doable, you just need to recover from the second process and perform the iteration there. To refresh the latest updates, you would need to "recover" again to the latest commit.

[AlgorithmsAreCool]

That is ideal! I needed to be able to progressively read from the log and was hoping for something like this.

Just to be clear, I'm assuming that truncating or writing from the 2nd process is not supported, correct?

[badrishc]

To be clear, the basic underlying capability is there but is not exposed or supported out of the box right now. I tried to play around and it works but will need some effort to convert into a supported feature.

Work in branch: fasterlog-separate-scan
See working sample here: https://github.com/microsoft/FASTER/blob/fasterlog-separate-scan/cs/samples/FasterLogPubSub/Program.cs

[badrishc]

I'm assuming that truncating or writing from the 2nd process is not supported, correct?

That is correct. The reader gets a read-only view of the log.

[badrishc]

Marking this as feature request/enhancement based on this work.

[badrishc]

It's now in relatively stable state - but still in branch. Feel free to play with the sample to see if that is what you had in mind. Log truncation right now needs to be handled by the primary owner (writer), as it is the point of truth for the log.

The way we expose the reading capability in the second process is summarized in the code snippet below:

static async Task SeparateConsumerAsync(CancellationToken cancellationToken)
{
    var device = Devices.CreateLogDevice(path + "mylog");
    var log = new FasterLog(new FasterLogSettings { LogDevice = device, ReadOnlyMode = true, PageSizeBits = 9, SegmentSizeBits = 9 });
    var _ = RecoverAsync(log, cancellationToken);

    using var iter = log.Scan(log.BeginAddress, long.MaxValue);

    await foreach (var (result, length, currentAddress, nextAddress) in iter.GetAsyncEnumerable(cancellationToken))
    {
        Console.WriteLine($"Consuming {Encoding.UTF8.GetString(result)}");
        iter.CompleteUntil(nextAddress);
    }
}

static async Task RecoverAsync(FasterLog log, CancellationToken cancellationToken)
{
    while (!cancellationToken.IsCancellationRequested)
    {
        await Task.Delay(TimeSpan.FromMilliseconds(restorePeriodMs), cancellationToken);

        Console.WriteLine("Restoring ...");

        log.RecoverReadOnly();
    }
}

[AlgorithmsAreCool]

Goodness you are fast 🤯

I'm looking forward to seeing saturating my disk IO testing this out!

[badrishc]

The linked sample will probably have overheads due to string encoding and decoding. Using byte arrays or spans as in this sample is best for perf/benchmarking.